FLOSS Project Planets

Lullabot: Announcing The New Lullabot.com

Planet Drupal - Fri, 2015-07-31 15:05
Mmmm… love that new website smell! Some history

It's been nearly 10 years since we launched our first company website at lullabot.com. During that time, we've done five full redesigns of the site. The company has grown from two people to 62. We've expanded from a small Drupal consulting and education company to a full-service agency with a complete Design team, dedicated front-end developers, and of course, the expert Drupal back-end development which has always been our foundation.

As we've grown, our site design has reflected our focus and skills. The first site that Matt and I put together back in 2005 was intentionally sparse – not exactly beautiful, but functional and simple to maintain for just 2 or 3 people. As we hired talented designers and skilled front-end developers, site redesigns became more complex. In 2010, we split our Drupal education services into Drupalize.Me and the main focus of lullabot.com became our client services work, showcasing our design and development projects and sharing insights from our team.

Revving up the new Lullabot.com

The newest iteration of Lullabot.com is our most ambitious to date. As with most of our client engagements, the project started with research. Our Design team interviewed existing and potential clients, site visitors, and the Lullabot team to understand how people were using our site – what they wanted to get out of it, and why they visited. Our team distilled all they'd learned into goals and early wireframes for the site. They then worked with our Development staff to try to come up with the most flexible way of achieving these goals so that we could have full control of the site in ways that Drupal often doesn't afford. They wanted full <html> to </html> blue-sky design of any arbitrary page on the site without losing Drupal's amazing content management capabilities.

The technical team settled on a decoupled, isomorphic approach using Facebook's React, Node.js, CouchDB (a noSQL database) and Drupal as the backend CMS.

Content management is what Drupal does best, and this happens through a purpose-built subsite where the Lullabot team can login and post articles, podcasts, and manage their bios. Drupal pushes content into CouchDB, which exposes a REST API for React to consume. React is an isomorphic library (its code can run both in the server and the client), which means that when a visitor first visits the site, they receive the html of the entire page. Then, the rest of the navigation happens client-side, updating just the parts of the page which are different from the current one. Furthermore, React is written to be completely backward compatible with older browsers.

Our clients are often in need of API-driven native mobile apps, television-based apps, and content ingestion on connected devices. We've implemented these things in less holistic ways with our clients in the past. But the new Lullabot.com gave us a chance to experiment with some methodologies that weren't quite tried-and-tested enough to recommend to our clients. But now that we've had a chance to see the type of flexibility they give us on lullabot.com, we'll be adding this to the array of architectural strategies that we can consider for our clients in the future.

Look ma, no hands!

The results are amazing; high-speed, high-performance, and superlative flexibility. In layman's terms, this means our Design and Front-end people can go crazy – implementing blue-sky ideas without the usual Drupal markup constraints. The new site is fully responsive. Articles and portfolio work pages can have giant, dazzling, full browser-height background images or videos. Articles have big text that is easy to read on any scale from large desktop monitors to the smallest phone screens. Furthermore, we did everything with an eye toward blazing fast page loads. We omitted jQuery, trading convenience in the development process for speedy page loads. Then we looked at every http request, every image, every library to make sure our website was as snappy on an older smartphone as it was on the desktop. Best of all, we off-loaded much of the heavy lifting to the client-side with React.

Design-wise, the new site is uncluttered, sparse, and relatively simple. But whether you're looking for our vast archive of articles or podcasts, information about what services Lullabot offers, who we've worked with and what we've done, or you're curious to know what it's like to work at Lullabot, it's all there.

Over the coming months, we will be writing a series of articles and doing a few podcasts talking about different aspects of the new site. Please subscribe to the Lullabot email newsletter below and you'll be the first to know when new articles are published.

Categories: FLOSS Project Planets

It’s the final countdown

Planet KDE - Fri, 2015-07-31 14:21

As my time in Spain in coming to a close I reflect on my time here and at Akademy. I have learned that KDE is even more awesome and the community is more amazing then I ever thought (this was my first Akademy). I’ve done quite a few things during my time here, from coding (QML), riding a Paddle Board, to packaging and debugging. I’ve seen time and time again that the people who work in Open Source are some of the nicest people you can find anywhere.

Thanks are in order

Categories: FLOSS Project Planets

Midwestern Mac, LLC: Launching my first Drupal 8 website — in my basement!

Planet Drupal - Fri, 2015-07-31 13:48

I've been working with Drupal 8 for a long time, keeping Honeypot and some other modules up to date, and doing some dry-runs of migrating a few smaller sites from Drupal 7 to Drupal 8, just to hone my D8 familiarity.

I finally launched a 'for real' Drupal 8 site, which is currently running on Drupal 8 HEAD—on a cluster of Raspberry Pi 2 computers in my basement! You can view the site at http://www.pidramble.com/, and I've already started posting some articles about running Drupal 8 on the servers, how I built the cluster, some of the limitations of at-home webhosting, etc.

Categories: FLOSS Project Planets

OSTraining: Image Optimization with WebResizer.com and ImageRecycle.com

Planet Drupal - Fri, 2015-07-31 13:15

One of the biggest factors to a slow page load is unoptimized image sizes. The bigger the image, the more time it takes the browser to load it.

This tutorial will show you the two free tools that I use to optimize images quickly.

These services will be useful whether you use WordPress, Drupal, Joomla or any other platform.

Categories: FLOSS Project Planets

A shout-out to SQLAlchemy

Planet KDE - Fri, 2015-07-31 12:33

At work-work we write Python apps, using SQLAlchemy as ORM and database layer. I tend to run our stuff on OpenSUSE with SQLite for quick development purposes, but production tends to be MS SQL Server and Windows XP (there are migrations ongoing ..). Today we got a bug report that certain columns in the database were being created as VARCHAR(max) (that’s a SQL-Serverism) and that this was leading to errors elsewhere in the application when reflection of the database was used to (re-)retrieve the types of columns.

So I spent a few hours testing and writing up a careful bug report with a reproducible test case. That’s the time my job is (most) fun, writing up something that contributes outside our own walls.

Turns out we were doing something that wasn’t quite right, but the SQLAlchemy developers agreed that it was a not-unreasonable thing to do and within six hours there was a fix, committed and scheduled for the next release. So here’s a shout-out to Mike B. at SQLAlchemy for his quick work. (And I’m glad the effort of making a good-as-I-can bugreport paid off.)

Categories: FLOSS Project Planets

Steve McIntyre: Linaro VLANd v0.3

Planet Debian - Fri, 2015-07-31 12:04

VLANd is a python program intended to make it easy to manage port-based VLAN setups across multiple switches in a network. It is designed to be vendor-agnostic, with a clean pluggable driver API to allow for a wide range of different switches to be controlled together.

There's more information in the README file. I've just released v0.3, with a lot of changes included since the last release:

  • Massive numbers of bugfixes and code cleanups
  • Added two new switch drivers:
    • TP-Link TL-SG2XXX family (TPLinkTLSG2XXX)
    • Netgear XSM family (NetgearXSM)
  • Added "debug" option to all the switch drivers to log all interactions
  • Added internal caching of port modes within the driver core for a large speed-up in normal use
  • Bug fix to handling of trunk ports in the CiscoCatalyst driver, improving VLAN interop with other switches
  • Huge changes to the test lab, now using 5 switches and 10 hosts
  • Big improvements to the test suite:
    • Match the new test lab layout
    • Move more of the core test code into the test-common utility library
    • Massively improved the check-networks test runner for the test hosts
    • Added parsing of the UP/DOWN results in test-common to give a simple PASS/FAIL result for each test
    • Added more tests
  • All logging now in UTC

VLANd is Free Software, released under the GPL version 2 (or any later version). For now, grab it from git; tarballs will be coming shortly.

Categories: FLOSS Project Planets

OSTraining: Googlebot Cannot Access CSS and JS Files

Planet Drupal - Fri, 2015-07-31 11:57

Many website owners have recently received an email from Google with the title, "Googlebot cannot access CSS and JS files".

It doesn't matter whether you're running WordPress, Drupal, Joomla or another platform entirely. Google has sent these emails to 100,000's of sites.

I'm going explain the issue that Google is complaining and how you can resolve it.

Categories: FLOSS Project Planets

Kubuntu Paddleboard Club

Planet KDE - Fri, 2015-07-31 11:30

I always say the best way to tour a city is from the water

Categories: FLOSS Project Planets

InternetDevels: Lviv Euro DrupalCamp 2015: the best choice for your awesome autumn!

Planet Drupal - Fri, 2015-07-31 10:57

Hi, drupalers! Enjoying your summer? Time to make plans for your awesome autumn! We have checked the forecasts, analyzed the position of stars, written php scripts to define the luckiest date and place for you. Here you go, the answer is: October 17-18, Lviv Euro DrupalCamp. The time and place cannot be changed ;)

Read more
Categories: FLOSS Project Planets

Rapha&#235;l Hertzog: My Free Software Activities in July 2015

Planet Debian - Fri, 2015-07-31 10:45

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it’s one of the best ways to find volunteers to work with me on projects that matter to me.

Debian LTS

This month I have been paid to work 15 hours on Debian LTS. In that time I did the following:

  • Finished the work on tracker.debian.org to make it display detailed security status on each supported release (example).
  • Prepared and released DLA-261-2 fixing a regression in the aptdaemon security update (happening only when you have python 2.5 installed).
  • Prepared and released DLA-272-1 fixing 3 CVE in python-django.
  • Prepared and released DLA-286-1 fixing 1 CVE in squid3. The patch was rather hard to backport. Thankfully upstream was very helpful, he reviewed and tested my patch.
  • Did one week of “LTS Frontdesk” with CVE triaging. I pushed 19 commits to the security tracker.
Kali Linux / Debian Stretch work

Kali Linux wants to experiment something close to Debian Constantly Usable Testing: we have a kali-rolling release that is based on Debian Testing and we want to take a new snapshot every 4 months (in order to have 3 releases per year).

More specifically we have a kali-dev repository which is exactly Debian Stretch + our own Kali packages (the kali package take precedence) updated 4 times a day, just like testing is. And we have a britney2 setup that generates kali-rolling out of kali-dev (without any requirement in terms of delay/RC bugs, it just ensures that dependencies are not broken), also 4 times a day.

We have jenkins job that ensures that our metapackages are installable in kali-dev (and kali-rolling) and that we can build our ISO images. When things break, I have to fix them and I try to fix them on the Debian side first. So here are some examples of stuff I did in response to various failures:

  • Reported #791588 on texinfo. It was missing a versioned dependency on tex-common and migrated too early. The package was uninstallable in testing for a few days.
  • Reported #791591 on pinba-engine-mysql-5.5: package was uninstallable (had to be rebuilt). It appeared on output files of our britney instance.
  • I made a non-maintainer upload (NMU) of chkrootkit to fix two RC bugs so that the package can go back to testing. The package is installed by our metapackages.
  • Reported #791647: debtags no longer supports “debtags update –local” (a feature that went away but that is used by Kali).
  • I made a NMU of debtags to fix a release critical bug (#791561 debtags: Missing dependency on python3-apt and python3-debian). kali-debtags was uninstallable because it calls debtags in its postinst.
  • Reported #791874 on python-guess-language: Please add a python 2 library package. We have that package in Kali and when I tried to sync it from Debian I broke something else in Kali which depends on the Python 2 version of the package.
  • I made a NMU of tcpick to fix a build failure with GCC5 so that the package could go back to testing (it’s part of our metapackages).
  • I requested a bin-NMU of jemalloc and a give-back of hiredis on powerpc in #792246 to fix #788591 (hiredis build failure on powerpc). I also downgraded the severity of #784768 to important so that the package could go back to testing. Hiredis is a dependency of OpenVAS and we need the package in testing.

If you analyze this list, you will see that a large part of the issues we had come down to package getting removed from testing due to RC bugs. We should be able to anticipate those issues and monitor the packages that have an impact on Kali. We will probably add new jenkins job that installs all the metapackages and then run how-can-i-help -s testing-autorm --old… I just submitted #794238 as a wishlist against how-can-i-help.

At the same time, there are bugs that make it into testing and that I fix / work around on the Kali side. But those fixes / work around might be more useful if they were pushed to testing via testing-proposed-updates. I tried to see whether other derivatives had similar needs to see if derivatives could join their efforts at this level but it does not look like so for now.

Last but not least, bugs reported on the Kali side also resulted in Debian improvements:

  • I reported #793360 on apt: APT::Never-MarkAuto-Sections not working as advertised. And I submitted a patch.
  • I orphaned dnswalk and made a QA upload to fix its only bug.
  • We wanted a newer version of the nvidia drivers. I filed #793079 requesting the new upstream release and the maintainer quickly uploaded it to experimental. I imported it on the Kali side but discovered that it was not working on i386 so I submitted #793160 with a patch.
  • I noticed that Kali build daemons tend to accumulate many /dev/shm mounts and tracked this down to schroot. I reported it as #793081.
Other Debian work

Sponsorship. I sponsored multiple packages for Daniel Stender who is packaging prospector, a software that I requested earlier (through RFP bug). So I reviewed and uploaded python-requirements-detector, python-setoptconf, pylint-celery and pylint-common. During a review I also discovered a nice bug in dh-python (#793609a comment in the middle of a Build-Depends could break a package). I also sponsored an upload of notmuch-addrlookup (new package requested by a Freexian customer).

Packaging. I uploaded python-django 1.7.9 in unstable and 1.8.3 in experimental to fix security issues. I uploaded a new upstream release of ditaa through a non-maintainer uploaded (again at the request of a Freexian customer).

Distro Tracker. Beside the work to integrate detailed security status, I fixed the code to be compatible with Django 1.8 and modified the tox configuration to ensure that the test suite is regularly run against Django 1.8. I also merged multiple patches of Christophe Siraut (cf #784151 and #754413).


See you next month for a new summary of my activities.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

Categories: FLOSS Project Planets

Mike Crittenden: Drupal 7 Deconstructed

Planet Drupal - Fri, 2015-07-31 09:40
How does Drupal work? Let's find out.

For years now I've wanted to dig through Drupal core, line by line, and understand how the big pieces do what they do. I'm finally doing that, and writing up my notes as I go.

Drupal 7 Deconstructed is the in-progress result of that.

If you've ever wondered what happens in the bootstrap process, or how Drupal's Form API works, or how exactly Drupal figures out which menu callback to run per page request, then this is the place to go.

It's just getting started, and so far I've only gone through the bootstrap process and the menu router, but I'm having a great time and learning a ton, so I expect to fill it up quickly.

Who could benefit from this?

Any developer who has ever wondered how Drupal works could get some value out of reading this. You'll need to know at least a little about Drupal development to understand parts (for example, I don't explain what hook_menu() is when talking about the menu router), but you shouldn't need to be an expert or anything.

If you feel like that describes you, but you don't understand a part, please let me know so that I can make it more approachable.

Want to help?

If you're interested in helping out, the best thing to do would be to keep an eye on the repo and proofread or review things as they're written.

Pull requests are also greatly appreciated, whether you want to fix a typo or submit a whole new chapter.

Or, if nothing else, just let me know if you like this idea! Knowing that this could be helpful to people besides just me is a huge motivational boost to keep things moving.

What's the end game?

I don't know. I could see this staying on GitHub forever, or being published on Leanpub, or ending up as a blog series.

Any suggestions?

Why Drupal 7? What about Drupal 8?

I chose Drupal 7 because it still has a pretty long shelf life left. Drupal 8 Deconstructed definitely needs to be written though, and I'd love to dive into that after 7 is complete.

What about contrib?

I would love to take apart some of the more commonly used contrib modules like Views, CTools, Panels, Webform, Pathauto, etc., as well, but one step at a time!

Please check out Drupal 7 Deconstructed and let me know what you think so far!

Read this next: I wrote a book for O'Reilly: "Responsive Theming for Drupal"
Categories: FLOSS Project Planets

Paul Bissex: Does Python scale?

Planet Python - Fri, 2015-07-31 06:47

(This is another thing I found myself writing on Quora and wanted to keep. The question was "Does Python have any scalability limitations?")

"Scalability" is a term people like to throw around, but the less specific you are as to what you mean by it, the less substantial the answers will be. It is not a simple linear measure on which languages can be given some numerical score.

Languages and their implementations do have certain inherent performance characteristics, but in order to understand their relevance to your needs you have to get specific about your needs.

You will always be able to find stories where people used technology X and found it "didn't scale". Sometimes this is because they didn't know, or chose not to pursue, certain performance optimizations. Sometimes this is because technology X was a poor fit for their problem.

(If you're into programming language esoterica, the technical feature of Python that gets the most attention in performance discussions is the Global Interpreter Lock.)

If it suits your project, use it. There's not some hidden performance ceiling that's going to suddenly appear and crush you. The Python system I work on serves 20+ million pages per day.

Categories: FLOSS Project Planets

Drupal core announcements: Recording from July 31st 2015 Drupal 8 critical issues discussion

Planet Drupal - Fri, 2015-07-31 06:29

We met again today to discuss critical issues blocking Drupal 8's release (candidate). (See all prior recordings). Here is the recording of the meeting video and chat from today in the hope that it helps more than just those who were on the meeting:

If you also have significant time to work on critical issues in Drupal 8 and we did not include you, let me know as soon as possible.

The meeting log is as follows (all times are CEST real time at the meeting):

[11:03am] jibran: I think it is sorted by name
[11:03am] jibran: the order in the hangout
[11:03am] WimLeers: y
[11:07am] jibran: We have to look at google hangout code base for that.
[11:08am] WimLeers: https://www.drupal.org/node/2499157#comment-10172426
[11:08am] Druplicon: https://www.drupal.org/node/2499157 => [meta] Auto-placeholdering [#2499157] => 5 comments, 4 IRC mentions
[11:11am] WimLeers: amateescu's issue link: https://www.drupal.org/node/2336627#comment-10160850
[11:11am] Druplicon: https://www.drupal.org/node/2336627 => Deadlock on cache_config (DatabaseBackend::setMultiple()) [#2336627] => 39 comments, 24 IRC mentions
[11:12am] WimLeers: plach: yay for vacation :D
[11:12am] GaborHojtsy: VACATIOOOOOON!
[11:12am] GaborHojtsy: sometime, sometime :)
[11:12am] alexpott: https://www.drupal.org/node/2542762 is the nearly ready issue
[11:12am] Druplicon: https://www.drupal.org/node/2542762 => hook_entity_type_update doesn't get the entity in the new revision after addTranslation and setNewRevision [#2542762] => 11 comments, 4 IRC mentions
[11:13am] alexpott: https://www.drupal.org/node/2542748 is the gnarly update issue
[11:13am] Druplicon: https://www.drupal.org/node/2542748 => Automatic entity updates are not safe to run on update.php by default [#2542748] => 21 comments, 7 IRC mentions
[11:15am] plach: WimLeers: :)
[11:15am] WimLeers: alexpott: yay :)
[11:15am] dawehner: https://www.drupal.org/node/2540416
[11:15am] Druplicon: https://www.drupal.org/node/2540416 => Decide whether we need hook_upgrade_N()/upgrade.php front controller [#2540416] => 27 comments, 4 IRC mentions
[11:16am] WimLeers: dawehner: cache tables are auto-created
[11:16am] naveenvalecha|af left the chat room. (Read error: Connection reset by peer)
[11:16am] WimLeers: but yeah, router table etc… #sadpanda
[11:19am] WimLeers: The issue that originally turned it from a separate PHP file into a route + controller: https://www.drupal.org/node/2250119
[11:19am] Druplicon: https://www.drupal.org/node/2250119 => Run updates in a full environment [#2250119] => 21 comments, 1 IRC mention
[11:34am] WimLeers: plach: ROFL
[11:34am] WimLeers: plach++
[11:43am] WimLeers: "a foam of circles" lol
[11:48am] alexpott: https://www.drupal.org/node/2542748
[11:48am] Druplicon: https://www.drupal.org/node/2542748 => Automatic entity updates are not safe to run on update.php by default [#2542748] => 21 comments, 8 IRC mentions
[11:55am] dawehner: WimLeers: well but those tables aren't auto fixed
[11:56am] dawehner: WimLeers: so just imagine what happens if you need to change the cache_ tables
[12:00pm] WimLeers: dawehner: ohhh!
[12:02pm] WimLeers: plach: is that the church bells in Venice that I'm hearing?
[12:03pm] WimLeers: dawehner: lol
[12:03pm] WimLeers: :)
[12:05pm] plach: WimLeers: yeah, sorry :)
[12:05pm] WimLeers: plach: made me feel like I was on vacation, ever so briefly
[12:05pm] WimLeers: :D
[12:05pm] plach: :)
[12:23pm] dawehner: alexpott: are the issues the new thing or the solutions ;)
[12:24pm] alexpott: dawehner: well we have better ideas

Categories: FLOSS Project Planets

Akademy 2015 & Kate

Planet KDE - Fri, 2015-07-31 04:53

Today, I will travel back home from Akademy 2015.

I must say, it was really a nice KDE meeting and I had a lot of fun ;=)
The first day the KDE e.V. general assembly did take place, then two days of actually interesting talks (including the great announcement of Plasma Mobile). After some more days with interesting BoFs and hacking, Akademy is now ending for me.

I didn’t do that much work on Kate, I mostly did small bugfixes for the applications bundled with the KDE Applications releases regarding their HiDPI support, finally no Konsole that can’t redraw correctly on scrolling on a HiDPI screen with scaling activated!

For Kate, the most stuff I did was porting one more plugin (the text filter plugin), fixing some small bugs and rearranging the search bars for in document and in files search. I hope it now is a bit nicer to use, still that is not the final state and I guess we will ask the VDG for more input later on. (btw., if you see any HiDPI glitches in Kate/KTextEditor master, please inform me, I really want to have non-pixelated output ;=)

A big thanks to all organizers & helpers of Akademy 2015! You did a great job, it was a lot of fun, the location was nice, the social event + day trip was good, all fine One of the nicer Akademy experiences! Lets hope that the flight home works out, as Dominik is taking the same plane, otherwise the Kate workforce will be seriously diminished.

Categories: FLOSS Project Planets

Simon Kainz: DUCK challenge: week 4

Planet Debian - Fri, 2015-07-31 03:15

The DUCK challenge is making a quite stable progress: in the last 4 weeks there were approximately 12.25 packages fixed and uploaded per week. In the current week the following packages were fixed and uploaded into unstable:

So we had 14 packages fixed and uploaded by 10 different uploaders. A big "Thank You" to you!!

Since the start of this challenge, a total of 49 packages, uploaded by 31 different persons were fixed.

Here is a quick overview:

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 # Packages 10 15 10 14 - - - Total 10 25 35 49 - - -

The list of the fixed and updated packages is availabe here. I will try to update this ~daily. If I missed one of your uploads, please drop me a line.

DebConf15 is approaching quite fast, so please get involved: The DUCK Challenge is running until end of DebConf15!

Pevious articles are here: Week 1, Week 2, Week 3.

Categories: FLOSS Project Planets

OpenStack Summit Tokyo 2015: Vote for Ceph Presentations

Planet KDE - Fri, 2015-07-31 02:24
In a few hours the voting period for the next OpenStack Summit in Tokyo ends at 30. July, 11:59 PDT (31. July  6:59 UTC / 08:59 CEST).
Here a some interesting talks regarding Ceph you may would like to vote for (this list is not intended to be exhaustive):
Categories: FLOSS Project Planets

Plasma Mobile References Images by Kubuntu

Planet KDE - Thu, 2015-07-30 21:32

We launched Plasma Mobile at KDE’s Akademy conference, a free, open and community made mobile platform.

Kubuntu has made some reference images which can be installed on a Nexus 5 phone.

More information is on the Plasma Mobile wiki pages.

Reporting includes:

Categories: FLOSS Project Planets

Justin Mason: Links for 2015-07-30

Planet Apache - Thu, 2015-07-30 19:58
  • danilop/yas3fs · GitHub

    YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3. It was inspired by s3fs but rewritten from scratch to implement a distributed cache synchronized by Amazon SNS notifications. A web console is provided to easily monitor the nodes of a cluster.

    (tags: aws s3 s3fs yas3fs filesystems fuse sns)

  • danilop/runjop · GitHub

    RunJOP (Run Just Once Please) is a distributed execution framework to run a command (i.e. a job) only once in a group of servers [built using AWS DynamoDB and S3]. nifty! Distributed cron is pretty easy when you’ve got Dynamo doing the heavy lifting.

    (tags: dynamodb cron distributed-cron scheduling runjop danilop hacks aws ops)

Categories: FLOSS Project Planets

Edward J. Yoon: Google's DistBelief Clone Project on Apache Hama

Planet Apache - Thu, 2015-07-30 19:29

 Deep Learning has become a household buzzword these days. Google, Microsoft, and Tencent have developed distributed deep learning systems but, these systems are closed source softwares. Many of open source softwares such as DeepDist, Caffe, ..., etc are data parallel only. In this blog post, I introduce an Artificial Neural Network implementation of Apache Hama ML package and future design plan for supporting both data and model parallelism.

1. Artificial Neural Network of Hama ML Package The lastest Apache Hama provides distributed training of an Artificial Neural Network using its BSP computing engine (the initial code was contributed by Yexi Jiang, a Hama committer, Facebook). In general, the training data is stored in HDFS and is distributed in multiple machines. In Hama, two kinds of components are involved in the training procedure: the master task and the groom task. The master task is in charge of merging the model updating information and sending model updating information to all the groom tasks. The groom tasks is in charge of calculate the weight updates according to the training data.

 The training procedure is iterative and each iteration consists of two phases: update weights and merge update. In the update weights phase, each groom task would first update the local model according to the received message from the master task. Then they would compute the weight updates locally with assigned data partitions (mini-batch SGD) and finally send the updated weights to the master task. In the merge update phase, the master task would update the model according to the messages received from the groom tasks. Then it would distribute the updated model to all groom tasks. The two phases will repeat alternatively until the termination condition is met (reach a specified number of iterations).

 The model is designed in a hierarchical way. The base class is more abstract than the derived class, so that the structure of the ANN model can be freely set by the user, as long as it is a layered model. Therefore, the Perceptron, Auto-encoder, Linear and Logistic regressor can all be uniformly represented by an ANN.

2. Future Plan for Large Scale Deep Neural Network2.1 Architecture As described in above section, currently the data parallelism is only used. Each node will have a copy of the model. In each iteration, the computation is conducted on each node and a final aggregation is conducted in one node. Then the updated model will be synchronized to each node. So, the performance is one thing; the parameters should fit into the memory of a single machine.

 Here is a tentative near future plan I propose for applications needing large model with huge memory consumptions, moderate computational power for one mini-batch, and lots of training data. The main idea is use of Parameter Server to parallelize model creation and distribute training across machines. Apache Hama framework assigns each split of training data stored in HDFS to each BSP task. Then, the BSP task assigns each of the N threads a small portion of work, much smaller than 1/Nth of the total size of a mini-batch, and assigns new portions whenever they are free. With this approach, faster threads do more work than slower threads:

 Each thread asynchronously asks the Parameter Server who stores the parameters in distributed machines for an updated copy of its model, computes the gradients on the assigned data, and sends updated gradients back to the parameter server. This architecture is inspired by Google's DistBelief (Jeff Dean et al, 2012).
2.2 Neuron-centric programming model The new Programming API proposed by Edward J. Yoon will provide two user-defined functions, which the user can define the characteristic of artificial neural network model: Activation function and Cost function.

 Each function can be implemented by extending ActivationFunction and CostFunction abstract classes like below:

* User-defined sigmoid actiavation function
public static class Sigmoid extends ActivationFunction {
public double apply(double input) {
return 1.0 / (1 + Math.exp(-input));

public double applyDerivative(double input) {
return input * (1 - input);

 The following properties are specified in the Job configuration:

  • The model topology: including the number of neurons (besides the bias neuron) in each layer; the type of squashing function; the degree of parallelization for each layer. 
  • The learning rate: Specify how aggressive the model learning the training instances. A large value can accelerate the learning process but decrease the chance of model convergence. Recommend in range (0, 0.5]. 
  • The momemtum weight: Similar to learning rate, a large momemtum weight can accelerate the learning process but decrease the chance of model convergence. Recommend in range (0, 0.5]. 
  • The regularization weight: A large value can decrease the variance of the model but increase the bias at the same time. As this parameter is sensitive, it's better to set it as a very small value, say, 0.001.

 The following is the sample design regarding how to create a job for training of three-layer neural network:

public static void main(String[] args) throws Exception {
ANNJob ann = new ANNJob();

// set learning rate and momentum weight

// initialize the topology of the model
// set the activation function and parallel degree.
ann.addLayer(featureDimension, Sigmoid.class, numOfTasks);
ann.addLayer(featureDimension, Sigmoid.class, numOfTasks);
ann.addLayer(labelDimension, Sigmoid.class, numOfTasks);

// set the cost function to evaluate the error

 In closing this blog post, I would like to request your feedback about design ideas. Please feel free to drop a comment or send a mail to our dev@hama.apache.org mailing list. Thanks.
Categories: FLOSS Project Planets

OSTraining: Using Theme Developer and Firebug to Debug Drupal Themes

Planet Drupal - Thu, 2015-07-30 18:47

There are several tools that that makes designing for Drupal much, much easier.

In particular, we recommend the Theme developer module and also Firebug.

Many of you may know about Firebug already. It's a browser tool that allows you to inspect any CSS, HTML or JavaScript elements.

Think of Theme developer as a Drupal-specific version of Firebug. Using Theme developer you can click on any element of your Drupal site and get a breakdown of how it was built.

Categories: FLOSS Project Planets
Syndicate content