FLOSS Project Planets

Wunderkraut blog: Just get the files you need when you develop

Planet Drupal - Wed, 2014-04-23 09:34

When you work locally on development or test on stage/dev whatever you sometimes needs the files from production. Our old way in solving that is downloading the whole file catalogue and have it local. Sometimes the file catalogues where several gigabytes large so that is not a good workflow at all.

To solve that problem we are now using Stage file proxy. We have been using it for some time now to get files to stage or locally, and it works really well (we have some issues on D6-sites, but works almost flawless on D7-sites). Stage file proxy downloads the files that are requested from production (or whatever) site running environment.

So you just get the files you need, and you could easily delete the files locally and get them back when called for. Time and space saver.

A nice patch to the module is getting an admin-interface - so that you don’t need to add settings to settings.php or creating variables. EDIT: Greggles just committed that patch to dev :-)

Photo by James Butler

Categories: FLOSS Project Planets

Petter Reinholdtsen: Install hardware dependent packages using tasksel (Isenkram 0.7)

Planet Debian - Wed, 2014-04-23 08:50

It would be nice if it was easier in Debian to get all the hardware related packages relevant for the computer installed automatically. So I implemented one, using my Isenkram package. To use it, install the tasksel and isenkram packages and run tasksel as user root. You should be presented with a new option, "Hardware specific packages (autodetected by isenkram)". When you select it, tasksel will install the packages isenkram claim is fit for the current hardware, hot pluggable or not.

The implementation is in two files, one is the tasksel menu entry description, and the other is the script used to extract the list of packages to install. The first part is in /usr/share/tasksel/descs/isenkram.desc and look like this:

Task: isenkram Section: hardware Description: Hardware specific packages (autodetected by isenkram) Based on the detected hardware various hardware specific packages are proposed. Test-new-install: mark show Relevance: 8 Packages: for-current-hardware

The second part is in /usr/lib/tasksel/packages/for-current-hardware and look like this:

#!/bin/sh # ( isenkram-lookup isenkram-autoinstall-firmware -l ) | sort -u

All in all, a very short and simple implementation making it trivial to install the hardware dependent package we all may want to have installed on our machines. I've not been able to find a way to get tasksel to tell you exactly which packages it plan to install before doing the installation. So if you are curious or careful, check the output from the isenkram-* command line tools first.

The information about which packages are handling which hardware is fetched either from the isenkram package itself in /usr/share/isenkram/, from git.debian.org or from the APT package database (using the Modaliases header). The APT package database parsing have caused a nasty resource leak in the isenkram daemon (bugs #719837 and #730704). The cause is in the python-apt code (bug #745487), but using a workaround I was able to get rid of the file descriptor leak and reduce the memory leak from ~30 MiB per hardware detection down to around 2 MiB per hardware detection. It should make the desktop daemon a lot more useful. The fix is in version 0.7 uploaded to unstable today.

I believe the current way of mapping hardware to packages in Isenkram is is a good draft, but in the future I expect isenkram to use the AppStream data source for this. A proposal for getting proper AppStream support into Debian is floating around as DEP-11, and GSoC project will take place this summer to improve the situation. I look forward to seeing the result, and welcome patches for isenkram to start using the information when it is ready.

If you want your package to map to some specific hardware, either add a "Xb-Modaliases" header to your control file like I did in the pymissile package or submit a bug report with the details to the isenkram package. See also all my blog posts tagged isenkram for details on the notation. I expect the information will be migrated to AppStream eventually, but for the moment I got no better place to store it.

Categories: FLOSS Project Planets

Frederick Giasson: Exporting Entities using OSF for Drupal (Screencast)

Planet Drupal - Wed, 2014-04-23 08:41

This screencast will introduce you to the OSF for Drupal features that let you export Drupal Entities in one of the following supported serializations:

  • RDF+XML (RDF in XML)
  • RDF+N3 (RDF in N3)
  • structJSON (Internal OSF RDF serialization in JSON)
  • structXML (Internal OSF RDF serialization in XML)
  • ironJSON (irON serialization in JSON)
  • commON (CSV serialization to be used in spreadsheet applications)

I will show you how you can use OSF for Drupal to export entire datasets of Entities, or how to export Entities individually. You will see how you can configure Drupal such that different users roles get access to these functionalities.

I will also briefly discuss how you can create new converters to support more data formats.

Finally, I will show you how Drupal can be used as a linked data platform with a feature that makes every Drupal Entities dereferencable on the Web1. You will see how you can use cURL to export the Entities‘ descriptions using their URI in one of the 6 supported serialization formats.


  1. OSF for Drupal follows the Cool URIs for the Semantic Web W3C’s interest group notes
Categories: FLOSS Project Planets

Drupal Commerce: Commerce Module Tuesday: Commerce Add to Cart Extras

Planet Drupal - Wed, 2014-04-23 08:35

Commerce Add to Cart Extras is a great little module that provides the ability to turn a product listing view into an add to cart form. It does this by providing a new field that you add to a view which provides a quantity text box and makes the entire view an add to cart form. This differs from the existing “Add to Cart” field provided by commerce, which allows you to add a single item to the cart from a list.

This allows you to do some useful things such as creating a bulk order form which lists some or all of the products on your site, allowing customers to enter a quantity directly for the products they want. Or, you can utilize standard Views functionality to create another add to cart form for wholesalers on products with several attributes allowing them to add quantities of each individual product quickly and easily. Let’s take a brief look at how we can do this.  Just download and enable the module, and you’re ready to get started.

Video and run-down after the break.

Categories: FLOSS Project Planets

Gergely Nagy: GSoC2014: syslog-ng accepted projects

Planet Debian - Wed, 2014-04-23 08:01

The Google Summer of Code 2014 programme reached another milestone recently: the accepted proposals were published, and over a thousand students were selected by nearly two hundred mentoring organisations, among them the syslog-ng project. We will be working with four students this year (twice we worked with last year), with more mentors, on a wider selection of projects. It was a tough decision to select the proposals, we received some very strong work this year.

We would like to express our gratitude to both Debian, for giving us an extra slot (so we could accept four students instead of three), and Google and Carol Smith in particular, for allowing it, and putting up with our fumbling during the process.

The accepted projects and students are:

Good luck to all students, we're looking forward to working with you throughout the summer! Happy hacking!

Categories: FLOSS Project Planets

Colm O hEigeartaigh: Apache Santuario - XML Security for Java 2.0.0 - part II

Planet Apache - Wed, 2014-04-23 07:26
In the previous blog post, I covered the new StAX-based (streaming) XML Signature functionality coming in Apache Santuario - XML Security for Java 2.0.0. In this post, I will focus on the new streaming XML Encryption functionality that will also be available in this release.

1) XML Encryption test-cases

I have uploaded some test-cases to github to show how to use the new StAX-based API. The tests and setup mirror the XML Signature testcases that I covered in the previous blog post. There are currently three junit tests in this project:

2) Outbound StAX configuration

To see how to configure the new outbound StAX-based XML Encryption functionality, take a look at the "encryptUsingStax" method used by the tests. The streaming XML Security functionality is configured by populating a XMLSecurityProperties Object. You must typically call the following methods for XML Encryption:
  • properties.setAction(XMLSecurityConstants.Action) - an "Action" to perform, which for XML Encryption purposes is XMLSecurityConstants.ENCRYPT. 
  • properties.setEncryptionKey(Key) - The encrypting key. Typically a SecretKey instance.
  • properties.setEncryptionSymAlgorithm(String) - Symmetric encryption Algorithm to use. The default is AES 256.
  • properties.addEncryptionPart(SecurePart) - Add a SecurePart to encrypt, e.g. encrypt a given QName. 
  • properties.setEncryptionTransportKey(Key) - The key to use to encrypt the secret key (if desired). Either a SecretKey or PublicKey instance.
  • properties.setEncryptionKeyTransportAlgorithm(String) - The encryption key transport algorithm to use, to encrypt the secret key (if desired). Default is RSA OAEP.
  • properties.setEncryptionKeyIdentifier(SecurityTokenConstants.KeyIdentifier) - How to reference the encrypting key/cert. The default is SecurityTokenConstants.KeyIdentifier_IssuerSerial.  
The valid Encryption KeyIdentifier types are:
  • SecurityTokenConstants.KeyIdentifier_KeyValue
  • SecurityTokenConstants.KeyIdentifier_KeyName
  • SecurityTokenConstants.KeyIdentifier_IssuerSerial
  • SecurityTokenConstants.KeyIdentifier_SkiKeyIdentifier
  • SecurityTokenConstants.KeyIdentifier_X509KeyIdentifier
  • SecurityTokenConstants.KeyIdentifier_X509SubjectName
  • SecurityTokenConstants.KeyIdentifier_NoKeyInfo
3) Inbound StAX configuration

To see how to configure the new inbound StAX-based XML Encryption functionality, take a look at the "decryptUsingStAX" method used by the tests. As with encryption creation, it is necessary to create a XMLSecurityProperties Object, and to tell it what "Action" to perform. In addition you must call the following method:
  • properties.setDecryptionKey(Key) - a key to use to decrypt the request.
In addition, it is possible to pass a SecurityEventListener to record what security events were performed (see TestSecurityEventListener used in the tests). This allows the user to check what algorithms were used, what was encrypted etc.
Categories: FLOSS Project Planets

Music Production and Experimentation

LinuxPlanet - Wed, 2014-04-23 06:05
Of late, my interests have diversified into music production and composition. It's likely that over the next the next few blog posts I'll be detailing some of what I've been working on.
Sources for some of the better DAW software currently available:

Sources for some of the better VSTs that I've come across:

Manufacturers of music hardware:

Sources for songs and samples:

Sources for media and other resources:
Categories: FLOSS Project Planets

Henrik Sandklef: GNU Xnee among hotpicks :)

GNU Planet! - Wed, 2014-04-23 04:10

GNU Xnee was listed among the HotPicks In Linux Format.



Categories: FLOSS Project Planets

Shai Erera: Updatable DocValues Under the Hood

Planet Apache - Wed, 2014-04-23 04:03
We recently committed updatable numeric and binary doc-values fields, which lets you update the value of such fields in a document without re-indexing the whole document. This allows using such fields for e.g. boosting documents relevancy to a query, especially when the boosting factor is frequently changing (like date, ratings, counters etc.). In this post I describe the under-the-hood mechanics of updating those fields.

A brief overview on DocValues

DocValues have been around since Lucene 4.0, and while they have gone through several changes and improvements since then, the basic concept remained the same – they let you record a value per-document per-field in each segment, and allow for very efficient random-access retrieval of those values. Lucene offers several doc-values types: numeric for single-valued numeric fields (date, ratings, counters), sorted/sortedset for single/multi-valued string (mostly, but can be any BytesRef) fields used for sorting, faceting, grouping etc., and binary fields for storing arbitrary byte[] per-document.

Also, Lucene provides several implementations for writing and reading DocValues: Lucene45 (the current default) stores the values on disk, but keeps some information in-memory for fast lookups. Memory loads the values into compressed arrays, while Direct loads the values into uncompressed arrays (faster lookups, but higher memory consumption). If your application runs in a resource constrained environment, you can use Disk which keeps absolutely everything on disk, at the cost of slower lookups.

DocValues record a value for every document in every field and segment even when a document has no value associated with a field. In most cases this is acceptable because you usually store a value for most, if not all, documents in a field (e.g. scoring factors, facets). But even if you index very sparse doc-values fields, the default encoding is very efficient at indexing blocks of "missing" values. Still, if you index hundreds of millions of documents with few dozen doc-values fields, but only a small percentage of them contain a value, writing your own sparse doc-values encoding may be beneficial.

Document deletions are a form of in-place document updates!

The mechanism by which Lucene updates DocValues fields is very similar to how document deletions are handled, so let’s look at how that’s done first. When you issue e.g. a deleteDocument(term) request, Lucene caches the request until an updated SegmentReader is needed, or the index is committed. It then resolves all the issued deletes to determine which documents are affected, both in already flushed segments as well as documents that were in IndexWriter’s RAM buffer when the delete was issued. Up-to-date SegmentReaders are needed in two cases: when a merge kicks off (SegmentMerger uses SegmentReaders to merge segments) and when the application reopens its NRT (near-real-time) reader. If an updated reader is needed, Lucene carries the deletes in memory by turning off a bit in the in-memory bit vector of each affected segment, and avoids flushing the bit vector to the index directory, which may reside on a slow storage device (e.g. NFS share). If the application also has an NRT reader open, the bit vector is cloned (because the app’s reader should not be made aware of the new deletes until it’s reopened), leading to double RAM consumption by those vectors. However, once that cost is paid once, it is "fixed" (1 bit/doc, or maxDoc/8 bytes overall), no matter how many documents are later deleted. Also, as soon as the application releases its NRT reader (usually after reopening it), one of the vectors is released and can be reclaimed by the GC.

Updatable DocValues

Like document deletions, when the application issues e.g. an updateNumericDocValue(term, field, value) request, Lucene caches the request until an updated reader is needed or the index is committed, at which point it resolves the updates against all affected documents. Unlike deletes though, doc-values consume much more RAM, which makes carrying the updates in-memory not feasible for many applications: numeric doc-values may potentially consume 8 bytes per-document, while binary values are usually even more expensive.

Therefore, when the updates are resolved, Lucene currently rewrites the entire field that was updated to the index directory. So for example, if you update a numeric doc-values field of a document in a 1 million documents segment, Lucene rewrites 1 million numeric values, or 8MB (at most). Because Lucene segments are write-once, the updated field is written to a gen’d doc-values file of the segment. A reader then reads the values of a field from the respective gen’d doc-values file. While this has a highish cost during indexing, it does not affect search at all, because the values are read from a single file, and often this is a tradeoff that most applications are willing to make.

To illustrate, the code below indexes several documents with "date" and "ratings" NumericDocValuesFields and then lists the files in the index directory:
IndexWriterConfig conf = new IndexWriterConfig(...);
conf.setCodec(new SimpleTextCodec());
IndexWriter writer = new IndexWriter(dir, conf);
ReaderManager manager = new ReaderManager(writer, true);
for (int i = 0; i < 5; i++) {
Document doc = new Document();
doc.add(new StringField("id", ""+i, Store.NO));
doc.add(new NumericDocValuesField("date", i));
doc.add(new NumericDocValuesField("ratings", i*10));

manager.maybeRefresh(); // triggers a flush

// update the 'date' field's value of document '0'
new Term("id", "0"), "date", System.currentTimeMillis());
manager.maybeRefresh(); // triggers applying the update
You will see the following print:

Notice how after the "date" field was updated and NRT reader re-opened, the index directory contains two sets of doc-values files: _0.dat stores both the "date" and "ratings" values and _0_1.dat stores just the "date" field’s values (the _1 part in the file name is the doc-values generation for the "date" field). Since the code uses SimpleTextCodec, we can easily print the contents of each file. Below is such print, clipped to show just the header of the files:
file: _0_1.dat
field date

file: _0.dat
field date
field ratings
If we update the value of the "date" field for the same or any other document and then list the doc-values files in the index, we will see:
Notice how _0_1.dat no longer exists in the index - that’s because it only contains values for the "date" field, which was entirely rewritten by the second update (under generation 2), and so _0_1.dat can be deleted.

Stacked Segments

On LUCENE-4258 we’ve started to implement field updates taking a stacked-segments approach. A stacked segment can be thought of as a layer/filter onto an existing segment, which indexes only the updates to that segment. When the full view of the segment is required (e.g. during search), the segment "collapses" all the updates down, so that the most up-to-date view of the segment is presented to the application.

The approach taken to implement updatable doc-values is similar in concept, only the doc-values "stacks" index the values of all documents. If we were to implement the approach taken on LUCENE-4258, then the doc-values stacks would index only the values of the updated documents. While this could have improved the indexing performance (writing a single value ought to be faster than writing 1 million values, no matter how fast your IO system is), it would affect search time since in order to resolve the value of a document, we would need to look for its most up-to-date value in any of the stacks. There are efficient ways to do this, but as usual, each comes with its own tradeoffs.

The key point is that stacked segments have a tradeoff between indexing and search and it is important to measure that tradeoff, as in Lucene, we often prefer to tradeoff in favor of search. I will publish in a separate post some benchmark numbers of the current approach, which will also serve as a baseline for measuring alternative approaches in the future.
Categories: FLOSS Project Planets

Andrew Pollock: [life] Day 85: Mostly a day off for me

Planet Debian - Wed, 2014-04-23 02:37

Zoe slept solidly all night. She woke up a little before 6am, wanting a cuddle, but it was still dark, so she went back to sleep for another half an hour or so. It was actually nice to get that extra 30 minutes to have a leisurely wake up myself.

Sarah wanted to pick up Zoe at 7:45am to get away and get a camp site, so when Zoe finally woke up for the day, we didn't muck around too much. She announced she wanted banana and oat pancakes, but we were out of bananas. I offered her the opportunity to scooter down to the Hawthorne Garage to get some if she went and got dressed. She decided that'd be good.

I had a 10am appointment in the city with an intellectual property lawyer to talk about patents, so I had this grand plan of trying to squeeze in a run after Zoe was picked up and before I'd have to head into the city, so I got into my running gear, and we headed down to acquire some bananas.

We whipped up the pancakes, and then after a couple of mouthfuls of hers, Zoe declared she didn't like it and wanted Cheerios instead. Oh well. It was nice to get out of the house early in the morning, and it helped get Zoe moving.

Sarah ended up getting here closer to 8:30am, which made it a little too tight to go for a run, have a shower and get into the city, so I scrapped the run and pottered around at home instead for a bit, before driving into the city.

My goodness casual parking in the city is exorbitant. It cost me $35 for under an hour. I got some good advice from the lawyer, so I know where to proceed from here.

Next I headed down to the Valley to get my orientation at River City Labs, but had I read my email before leaving the city, I'd have discovered that the lady giving me the orientation had to leave early because her daughter was ill. It cost me $6 drive into the car park in the Valley, take the elevator down, read my email on my phone and pay my ticket and leave again. Lesson learned.

I decided to do the grocery shopping that I hadn't done yesterday while I waited for Anshu to come over.

Categories: FLOSS Project Planets

Károly Négyesi: How to crash your site

Planet Drupal - Wed, 2014-04-23 02:14

I got a desperate call about a site being down, this is ordinary for me (advertisment: you can contact me if it happens to you). But the error I saw was new to me. This is surprising -- I have thought I have seen it all and then some. The modules/views/includes/plugins.inc was fataling about the function views_include not existing. At first I thought opcache went south cos how on earth could an include be loaded when the module isn't?? But it wasn't opcache. Next step was adding a debug_print_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS) before the offending views_include call and behold... what?? variable_initialize?? o_O OH! Obviously the poor thing is trying to unserialize a views object but it's superb early in the bootstrap and so modules aren't loaded. So while unserializing the classloader loads plugins.inc which leads to this fatal. Neat. Moral of the story: don't ever try to store a views object in variables. Or an array containing a views object.

Categories: FLOSS Project Planets

Gael Varoquaux: Google summer of code projects for scikit-learn

Planet Python - Wed, 2014-04-23 01:56

I’d like to welcome the four students that were accepted for the GSoC this year:

Welcome to all of you. Your submissions were excellent, and you demonstrated a good will to integrate in the project, with its social and coding dynamics. It is a privilege to work with you.

I’d also like to thank all the mentors, Alex, Arnaud, Daniel, James, Jaidev, Olivier, Robert and Vlad. It is a lot of work to mentor and mentors are not only making it possible for great code to enter scikit-learn, but also shaping a future generation of scikit-learn contributors.

Categories: FLOSS Project Planets

Europython: Financial assistance, partner programme, diversity and schedule

Planet Python - Wed, 2014-04-23 01:39

Financial Assistance Programme

In the second round of the EuroPython Financial Assistance Programme we granted an additional sum of 3000 EUR for April which results in a total sum  of 8000 EUR so far. There will be a third last round in May. So if you are seeking for financial support for coming to the EuroPython 2014 in Berlin you can still apply for the last round. 

We also started the sale of Financial Assistance support tickets last month. You can buy one or more Financial Assistance support tickets for a value of 10, 20, 50 or 100 Euro. With the purchase of  Financial Assistance tickets you will support further Python enthusiasts to go to EuroPython 2014 in case they need financial support for a ticket or travel expenses. The EuroPython organization team granted already 5.000 Euro last month in the first of three  rounds. Please show your appreciation for the Python community and help us to bring more people to the conference which could not make it otherwise.

Partner programme

We are currently working on the details for a partner & family programme in case you want to bring your better half or your whole family to Berlin. Berlin offers a lot of options for spending the time outside the conference programme like visits to museums, boat trips or guided trips to the Berliner “Unterwelten” and many other places of interest. A survey will be announced and published shortly in order to involve you and your partner into the arrangement and selection of the partner programme.

Diversity at EuroPython

There has been a heated discussion over the last week about the representation of women as speakers at EuroPython 2014 and about diversity in general. You can find the official thread here and the official statement of the EuroPython organizers here. Starting point of the discussion was that PyCon 2014 in Montreal reached an outstanding attendance of about  30% women (both as speaker and attendees).

The EuroPython organizers are committed to diversity by gender, sexual orientation and location. Besides the existing Code of Conduct we are working on an explicit Diversity Statement in order to make the EuroPython 2014 enjoyable for everyone. Making EuroPython more diverse is a long-time process that involves the EuroPython Society, the local organizers and the European Python community as a whole. Diversity and outreach can not be dictated but must be lived in reality on all levels.


Therehas been some confusion related to the published preliminary schedule (also under the aspects of diversity). The complete schedule will be announced shortly with roughly additional 50 talks and all trainings. The programme team is currently heavily working on the final version that will also take more diversity aspects into account for choosing the talks. We apologize for the confusion - specially to all persons that submitted a proposal keeping them in an uninformed state. The problem was caused by some internal miscommunication but also by the desire to publish a resilient schedule as early as possible in order to let the speakers make their travel arrangements in time.

Categories: FLOSS Project Planets

Modules Unraveled: 105 Using Membership Entity to Set Up a Drupal Based Membership Site with Caleb Thorne, Bryan Jones and David Csonka - Modules Unraveled Podcast

Planet Drupal - Wed, 2014-04-23 01:00
Published: Wed, 04/23/14Download this episodeProject
  • What is the Membership Entity module?
    Project background
    First membership site developed was Mercedes-Benz Club of America (http://www.mbca.org).
    Involved a bunch of custom code.
    Entity API
    Key features
    Multiple users per membership (primary and secondary)
    Join/Renew online
    Role management
    Membership terms and modifiers
    E-Commerce solutions (coming soon)
    Commerce integration (Caleb)
    Ubercart integration (Bryan)
    product class
    Using product attributes to store membership data.
    Investigating better ways to store data before official release.

  • Do you handle pro-rated renewals?

  • Trials?
  • How are you guys dealing with role based permissions, and time limited access?
  • Are there different levels of membership?
  • Do you deal with CC details?
  • What other solutions did you guys look at before creating this one?
    • Why not just use profile2/rules/civiCRM or other existing modules?

The big problem with Profile2 is that it only allows one user per profile. Some fields should be shared for all users that belong to a single membership.
Membership Entity includes rules integration.
Membership Entity is 100% Drupal. No third party integration required.

Use Cases
  • Do you guys have any sites currently running the Membership Entity module?
    (David) - Joined the team after much of the development of the module was already done, so had the perspective of a developer on the outside, learning what the module provides and how to extend it. Being built around the Entity API made creating Views-based reports very simple, as most of the membership data that I needed to display was already exposed. For any that wasn’t, the extensible nature of the module (via things like all of the provided custom hooks) made the process of developing additional custom entities derived from membership meta-data rather straightforward.
    (Bryan) - Drop in solution for simple membership sites (CEMA).
    No custom code required.
    Membership process working in less than an hour.
    Integrated with Ubercart (Module coming soon).
Episode Links: draenen (Caleb Thorne)BryanDavid CsonkaMonarch DigitalMembership EntityCommerce integration (sandbox)Ubercart integration (sandbox)Module release blog postWe will be having a BoF at DrupalCon Austin. Watch for it atTags: 
Categories: FLOSS Project Planets

Eko S. Wibowo: Reviving Microsoft Agent using PyWin32 - Run the Agent Hourly

Planet Python - Tue, 2014-04-22 20:00

Now the agent can run in a specified time interval

If you inspect closely our last article, you will realize that the Agent although having the ability to speak the time when you need it, it still unable to speak the time in specified interval, say, hourly. In this part of the article we are going to inspect whether we can easily add that ability.

Let's find out!

Quick Note: COM Threading Models

Long story short: Microsoft Agent is COM Object having Single Threading Apartment Model. You won't be able using MS Agent instance that have been instantiated in a different thread. For example, consider our current Agent application in previous article. As the Agent was instantiated in the main thread, seen below, 

1 2 3 4 if __name__ == '__main__':     ...     agent = MsAgent()     ...

You won't be able to use this agent object outside of this main thread. And why does it important? Because I want to display this Agent in a certain time interval (hourly, to be exact), meaning it must be run by a Python thread. The simple solution would be to instantiate and use this Agent entirely in another Thread, not separate its instantiation in main thread and use it in different thread. That's not going to work with COM Object having Single Threading Apartment Model as this MS Agent things.

How to run Python code in certain time interval?

To run Python code in certain time interval, you can easily use Python Timer class from threading module. The function below will run a thread that will wake up at exactly +1 o'clock from current time. 

1 2 3 4 5 6 7 8 9 10     def wakeup_next_hour(self):         '''        Run a thread that will wake up exactly n-o'clock        '''         now = datetime.datetime.now()         next_hour = now + datetime.timedelta(hours = 1)         next_hour_oclock = datetime.datetime(next_hour.year, next_hour.month, next_hour.day, next_hour.hour, 0, 0)         seconds = next_hour_oclock - now         self.thread = threading.Timer(seconds.total_seconds(), self.say_the_time_hourly)         self.thread.start()

Observe that, first, we get the value of what time is it now. And then, we calculate the next hour using timedelta from datetime module, later we calculate total seconds from current time until exactly next hour at H:0 o'clock. And finally, we run the thread using thread.start(). Don't forget to store this new thread object (using the statement self.thread = threading.Timer(..)) as object variable/property, for later dispose/cancellation of this running thread. 

The thread will run our new method say_the_time_hourly() that will call say_the_time() to  speak up the current time and schedule new thread for another hour. Below is the say_the_time_hourly() method:

1 2 3 4 5 6     def say_the_time_hourly(self):         '''        say the time and then schedule for another hour at exactly n-th o'clock        '''         self.say_the_time(None)         self.wakeup_next_hour()

Great. Now, what is the different between the previous say_the_time() method  with this new one? Let's inspect it in another section.

Safely Running MS Agent in Different Thread

As already been said, you can not use an instance of MsAgent that have been created from another thread. Previously, this is how we instantiate our MsAgent object:

1 2 3 4 5 6 7 8 9 10 11 12 13 class MsAgent(object):     def __init__(self):         self.agent=win32com.client.Dispatch('Agent.Control')         self.charId = 'James'         self.agent.Connected=1         self.agent.Characters.Load(self.charId)       def say_the_time(self, sysTrayIcon):         now = datetime.datetime.now()         str_now = '%s:%s:%s' % (now.hour, now.minute, now.second)         self.agent.Characters(self.charId).Show()         self.agent.Characters(self.charId).Speak('The time is %s' % str_now)         self.agent.Characters(self.charId).Hide()

If you don't modify the above method, and forcefully try to run say_the_time() from another Thread (using threading.Timer() object), Python will gives this exception: CoInitialize has not been called. This is a clue to indicate that to use COM object in a new thread, except that the object support multi-threading concurrency model, we must instantiate it here. Actually, Pythoncom, which is the core module of PyWin32 that dealing with any sort of Python <-> COM communication, silently call CoInitialize() when first imported in main thread. Therefore, if we would like to use COM object in different thread, we have to initialize it ourselves.

Using the above explanation, below is the code that will schedule the Agent to run hourly at exactly x-o'clock and speak up the time. Got to say, it is quite a fun!

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 class MsAgent(object):     '''    Construct an MS Agent object, display it and let it says the time exactly every hour    '''     def __init__(self):         self.charId = 'James'       def say_the_time(self, sysTrayIcon):         '''        Speak up the time!        '''         pythoncom.CoInitialize()         agent = win32com.client.Dispatch('Agent.Control')         charId = 'James'         agent.Connected=1         try:             agent.Characters.Load(charId)         except Exception as ex:             print ex           now = datetime.datetime.now()           if now.hour == 0 and now.minute == 0:             str_now = ' exactly %s o''clock' % now.hour         else:             str_now = '%s:%s:%s' % (now.hour, now.minute, now.second)         agent.Characters(charId).Show()           agent.Characters(charId).Speak('The time is %s' % str_now)         time.sleep(3)         agent.Characters(charId).Hide()         time.sleep(5)         agent.Characters.Unload(charId)         pythoncom.CoUninitialize()

Observe from the code above, that we initialize our MS Agent object completely in the function say_the_time(), that will be run by a thread, making it as a one way to a thread safe COM application. We begin and end the function by calling pythoncom.CoInitialize() and pythoncom.CoUninitialize() respectively, in order to have a clean instantiation and cleanup of our MS Agent object. Observe that we also pause the running code several seconds after each MS Agent animation, to let the animation finished. Without doing so, you won't see anything!

What's Next?

As always, you can get the current source code here: oclockreminder-version-2.zip.

Or, follow its public Github repository here: pythonthusiast/oclockreminder.

You maybe wondering, why I keep pursuing this application. "It is a dead technology!", as RJ implied in the comment section of the previous article. Well, it is. But I visioned that it will be a great starting point for a new productivity application that integrated seamlessly in a Windows environment!

What is it? 

Stay tuned for my next article!





Categories: FLOSS Project Planets

Justin Mason: Links for 2014-04-22

Planet Apache - Tue, 2014-04-22 19:58
Categories: FLOSS Project Planets

Montreal Python User Group: Python Project Night XII

Planet Python - Tue, 2014-04-22 19:42

Python Project Night XII After a much needed break from PyCon. We at Montréal-Python are eager to get into coding. Caravan Coop will be hosting the twelfth edition of Python Nights.

We will also invite a special guest project for people interested in helping out our community.

Please sign up on our Eventbrite event as the seats are limited: https://python-project-night-xii.eventbrite.ca

We would like to thank Caravan for hosting us this evening. See you there!

Categories: FLOSS Project Planets

Darren Mothersele: Drupal Site Builder Patterns - The State Machine

Planet Drupal - Tue, 2014-04-22 19:00

In this new series for my blog, I'll be documenting some common design patterns for Drupal site builds. This first post is about the State Machine pattern, which is something I've used on several sites recently.

First, let me explain what I mean by "pattern". If you are already familiar with design patterns in Object-oriented software then you can probably skip this bit, but I think it's useful for context.

Design patterns?

Here's a quote from the original Gang of Four book on design patterns. That book is about design of object-oriented software, but I think it applies to Drupal development too. The quote is from p.1 of the book, apologies if I offend anyone by bastardising it. I've taken the liberty of substituting the words "Designing object-oriented software" with "building Drupal sites", and a few other substitutions to make my point...

[Building large maintainable Drupal sites] is hard... Your design should be specific to the problem at hand but also general enough to address future problems and requirements [and be maintainable]... Experienced [Drupal site builders] will tell you that a reusable and flexible design is difficult if not impossible to get "right" the first time.

Yet experienced [Drupal site builders] do make good designs. Meanwhile new [site builders] are overwhelmed by the options available and tend to fall back on non-[Drupal] techniques they've used before. It takes a long time for novices to learn what good [Drupal site building] is all about. Experienced [site builders] evidently know something inexperienced ones don't. What is it?

One thing expert [site builders] know NOT to do is solve every problem from first principles. Rather, they reuse solutions that have worked for them in the past. When they find a good solution, they use it again and again. Such experience is part of what makes them experts.

So I've been looking at what these "good solutions" are that I might have been using, and as I identify them I've been documenting them along the same lines of the original design patterns from the Gang of Four book:

  • Pattern name - the handle we use to describe the problem
  • Problem - explain the problem and its context, and when you might want to use this pattern
  • Solution - describe the elements that make up the solution, in my case how the pattern can be best implemented in Drupal
  • Consequences - results and trade-offs of using the pattern, in this case I also consider further issues that many need to be considered as a result of using the pattern.

So, first let's look at what a state machine is, and what problems it solves, before going on to look at how to configure it in Drupal.

State Machine

A state machine is a theoretical computer science concept that provides a mathematical basis for modelling computation. But don't worry, the kind of state machines we'll be using don't require a degree in computer science to understand.

All you really need to know is that the state machine (or more correctly a Finite State Machine) has a finite number of "states" it can be in and the machine is only ever in one of these states at a time, it's current state. The state machine can change from one state to another triggered by an event or condition. This change of state is called a transition. A state machine is typically visualised using a state machine diagram, for example:

As you can see the states are represented by an ellipse with the name of the state inside, the arrows denote the possible transitions. You can also see how the entry point and exit point would be notated.

Here's a (very simplified) example of a ticket in an agile issue queue. In reality this would probably have several other states but for the sake of this example, here's a simple state machine for the ticket:

A state machine is defined by the list of possible states and the event/condition that triggers each transition.

If you're reading this and thinking "Events", "Conditions", sounds a bit like Drupal Rules, then you've already worked out how we're going to implement this in Drupal!

In this simple ticket example the states are "In progress", "Approval", and "Finished". The transitions are "Completed", "Rejected", "Accepted".

When to use it?

It might be useful to think that in business speak, when they say "business processes" they are actually talking about state machines. Here are some cases when you might want to think about state machines:

  • If you've ever had to model a "state" or "status" field, then you've got a good candidate for a state machine.
  • If you've ever wanted to anything more complex than just published and unpublished nodes then you have a good candidate for using a state machine.
  • If you have boolean fields in your content model called things like "paid/unpaid".
  • If you have records that need to expire after a specific period of time

Drawing out a state machine diagram to model this kinds of problems can be really useful to help identify any "edge-case" scenarios you may not have thought of, and capture them early in the design process. It also shows you exactly what you need to test further along in the site build.

Let's build it

As with anything in Drupal there are several ways to achieve this functionality, in fact there's even a State Machine module, but that relies on creating custom plugins. If you're a developer you might want to take a look at this module.

Workbench Moderation and various other workflow modules include a state machine implementation for a specific purpose.

The approach documented here is suitable for site builders, is flexible, and provides a neat solution that can be configured using the following contributed modules:

I said before that the state machine is defined by it's set of possible states and set of transitions. In Drupal we'll be using a simple list field to store the list of possible states for the node.

In a recent post on Drupalize.me they mention the addition of the ability to hide form fields in Drupal 8 core. In Drupal 7 we need a module to help us do this. In this case we are adding a field that will never be directly edited by the user so we just deny access to edit that field using the Field Permissions module.

For the simple ticket example, we have 3 states. So use an integer list field with the following allowed values:

  • 0|In progress
  • 1|Awaiting approval
  • 2|Finished

I said that the state machine was defined by the set of possible states (implemented by our list field), and a set of transitions. These transitions can be implemented using the Rules Link module.

Using the Rules Link module you can add a button to the ticket node which manipulates the "state" value preventing the user from actually editing the value in the state field directly, and thus enforcing the workflow defined in our state machine.

Each "Rules link" is configured in two parts. First you define the conditions for when the link should be visible using standard Rules conditions. Secondly, you use the rules reaction to set the value of the state field to the new value (and perform any other actions that you want as a side effect of the transition).


It's good to follow a principle of audit-ability, so you probably need to keep the transition history. A simple solution might be to add a timestamp field such as "confirmed at" to mark when it went to confirmed state. If using node, you could log revisions to track state changes in the revision log for the node. Or you could look at Messages module to log messages when state changes happen.

More patterns

If you're interested in learning more from my 7 years of Drupal experience (and if you're based in London) why not join me for Everything I Know About Drupal an intensive 2-day Drupal training I've been working on. It's taken a lot of preparation, and there's still a small number of tickets available. You can find more information on my blog post about it or grab a ticket on the Eventbrite page.

Categories: FLOSS Project Planets

Evolving Web: DrupalCamp NYC at the United Nations - Recap and Photos

Planet Drupal - Tue, 2014-04-22 17:52

This year's DrupalCamp NYC was held at the United Nations. The camp was crammed with summits and useful sessions and included a lot of content about Drupal 8.

read more
Categories: FLOSS Project Planets

Steve Kemp: I've not commented on security for a while

Planet Debian - Tue, 2014-04-22 17:14

Unless you've been living under a rock, or in a tent (which would make me slightly jealous) you'll have heard about the recent heartbleed attack many times by now.

The upshot of that attack is that lots of noise was made about hardening things, and there is now a new fork of openssl being developed. Many people have commented about "hardening Debian" in particular, as well as random musing on hardening software. One or two brave souls have even made noises about auditing code.

Once upon a time I tried to setup a project to audit Debian software. You can still see the Debian Security Audit Project webpages if you look hard enough for them.

What did I learn? There are tons of easy security bugs, but finding the hard ones is hard.

(If you get bored some time just pick your favourite Editor, which will be emacs, and look how /tmp is abused during the build-process or in random libraries such as tramp [ tramp-uudecode].)

These days I still poke at source code, and I still report bugs, but my enthusiasm has waned considerably. I tend to only commit to auditing a package if it is a new one I install in production, which limits my efforts considerably, but makes me feel like I'm not taking steps into the dark. It looks like I reported only three security isseus this year, and before that you have to go down to 2011 to find something I bothered to document.

What would I do if I had copious free time? I wouldn't audit code. Instead I'd write test-cases for code.

Many many large projects have rudimentary test-cases at best, and zero coverage at worse. I appreciate writing test-cases is hard, because lots of times it is hard to test things "for real". For example I once wrote a filesystem, using FUSE, there are some built-in unit-tests (I was pretty pleased with that, you could lauch the filesystem with a --test argument and it would invoke the unit-tests on itself. No separate steps, or source code required. If it was installed you could use it and you could test it in-situ). Beyond that I also put together a simple filesystem-stress script, which read/wrote/found random files, computes MD5 hashes of contents, etc. I've since seen similar random-filesystem-stresstest projects, and if they existed then I'd have used them. Testing filesystems is hard.

I've written kernel modules that have only a single implicit test case: It compiles. (OK that's harsh, I'd usually ensure the kernel didn't die when they were inserted, and that a new node in /dev appeared ;)

I've written a mail client, and beyond some trivial test-cases to prove my MIME-handling wasn't horrifically bad there are zero tests. How do you simulate all the mail that people will get, and the funky things they'll do with it?

But that said I'd suggest if you're keen, if you're eager, if you want internet-points, writing test-cases/test-harnesses would be more useful than randomly auditing source code.

Still what would I know, I don't even have a beard..

Categories: FLOSS Project Planets
Syndicate content