FLOSS Project Planets

April/May in KDE Itinerary

Planet KDE - Sat, 2019-06-01 04:00

A lot has happened again around KDE Itinerary since the last two month summary. A particular focus area at the moment is achieving “Akademy readiness”, that is being able to properly support trips to KDE Akademy in Milan early September, understanding the tickets of the Italian national railway is a first step into that direction.

New Features

The timeline view in KDE Itinerary now highlights the current element(s), to make it easier to find the relevant information. Active elements also got an “are we there yet?” indicator, a small bar showing the progress of the current leg of a trip, taking live data into account.

Trip element being highlighted and showing a progress indicator.

Another clearly visible addition can be found in the trip group summary elements in the timeline. Besides expanding or collapsing the trip, these elements now also show information concerning the entire trip when available, such as the weather forecast, or any power plug incompatibility you might encounter during the trip.

Trip group summary showing weather forecast for the entire trip. Trip group summary showing power plug compatibility warnings.

Less visible but much more relevant for “Akademy readiness” was adding support for Trenitalia tickets. That required some changes and additions to how we deal with barcodes, as well as an (ongoing) effort to decode the undocumented binary codes used on those tickets. More details can be found in a recent post on this subject.

Infrastructure Work

A lot has also happened behind the scenes:

  • The ongoing effort to promote KContacts and KCalCore yields some improvements we benefit from directly as well, such as the Unicode diacritic normalization applied during the country name detection in KContacts (reducing the database size and detecting country names also with slight spelling variations) or the refactoring of KCalCore::Person and KCalCore::Attendee (which will make those types easily accessible by extractor scripts).
  • The train reservation data model now contains the booked class, which is a particular useful information when you don’t have a seat reservation and need to pick the right compartment.
  • The RCT2 extractor (relevant e.g. for DSB, NS, ÖBB, SBB) got support for more variations of seat reservations and more importantly now preserves the ticket token for ticket validation with the mobile app.
  • The train station knowledge database is now also indexed by UIC station codes, which became necessary to support the Trenitalia tickets.
  • Extractor scripts got a new utility class for dealing with unaligned binary data in barcodes.

We also finally found the so far illusive mapping table for the station identifiers used in SNCF barcodes, provided by Trainline as Open Data. This yet has to find its way into Wikidata though, together with more UIC station codes for train stations in Italy. Help welcome :)

Performance Optimizations

Keeping an eye on performance while the system becomes more complex is always a good idea, and a few things have been addressed in this area too:

  • The barcode decoder so far was exposed more or less directly to the data extractors, resulting in possibly performing the expensive decoding work twice on the same document, e.g. when both the generic extractor and one or more custom extractors processed a PDF document. Additionally, each of those were applying their own heuristics and optimizations to avoid expensive decoding attempts where they are unlikely to succeed. Those optimizations now all moved to the barcode decoder directly, together with a positive and negative decoding result cache. That simplifies the code using this, and it speeds up extraction of PDF documents without a context (such as a sender address) by about 15%.
  • Kirigami’s theme color change compression got further optimized, which in the case of KDE Itinerary avoids the creation of a few hundred QTimer objects.
  • The compiled-in knowledge database got a more space-efficient structure for storing unaligned numeric values, cutting down the size of the 24bit wide IBNR and UIC station code indexes by 25%.
Fixes & Improvements

There’s plenty of smaller changes that are noteworthy too of course:

  • We fixed a corner case in KF5::Prison’s Aztec encoder that can trigger UIC 918.3 tickets, producing invalid barcodes.
  • The data extractors for Brussels Airlines, Deutsche Bahn and SNCF got fixes for various booking variants and corner cases.
  • Network coverage for KPublicTransport increased, including operators in Ireland, Poland, Sweden, parts of Australia and more areas in Germany.
  • More usage of emoji icons in KDE Itinerary got replaced by “real” icons, which fixes rendering glitches on Android and produces a more consistent look there.
  • Lock inhibition during barcode scanning now also works on Linux.
  • PkPass files are now correctly detected on Android again when opened as a content: URL.
  • The current trip group in the KDE Itinerary timeline is now always expanded, which fixes various confusions in the app when “now” or “today” don’t exist due to being in a collapsed time range.
  • Multi-day event reservations are now split in begin and end elements in the timeline as already done for hotel bookings.
  • Rental car bookings with a drop-off location different from the pick-up location are now treated as location changes in the timeline, which is relevant e.g. for the weather forecasts.
  • Extracting times from PkPass boarding passes now converts those to the correct timezone.

A big thanks to everyone who donated test data again, this continues to be essential for improving the data extraction.

If you want to help in other ways than donating test samples too, see our Phabricator workboard for what’s on the todo list, for coordinating work and for collecting ideas. For questions and suggestions, please feel free to join us on the KDE PIM mailing list or in the #kontact channel on Matrix or Freenode.

Categories: FLOSS Project Planets

Peter Bengtsson: Build an XML sitemap of XML sitemaps

Planet Python - Sat, 2019-06-01 01:06

Suppose that you have so many thousands of pages that you can't just create a single /sitemap.xml file that has all the URLs (aka <loc>) listed. Then you need to make a /sitemaps.xml that points to the other sitemap files. And if you're in the thousands, you'll need to gzip these files.

The blog post demonstrates how Song Search generates a sitemap file that points to 63 sitemap-{M}-{N}.xml.gz files which spans about 1,000,000 URLs. The context here is Python and the getting of the data is from Django. Python is pretty key here but if you have something other than Django, you can squint and mentally replace that with your own data mapper.

Generate the sitemap .xml.gz file(s)

Here's the core of the work. A generator function that takes a Django QuerySet instance (that is ordered and filtered!) and then starts generating etree trees and dumps them to disk with gzip.

import gzip from lxml import etree outfile = "sitemap-{start}-{end}.xml" batchsize = 40_000 def generate(self, qs, base_url, outfile, batchsize): # Use `.values` to make the query much faster qs = qs.values("name", "id", "artist_id", "language") def start(): return etree.Element( "urlset", xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" ) def close(root, filename): with gzip.open(filename, "wb") as f: f.write(b'<?xml version="1.0" encoding="utf-8"?>\n') f.write(etree.tostring(root, pretty_print=True)) root = filename = None count = 0 for song in qs.iterator(): if not count % batchsize: if filename: # not the very first loop close(root, filename) yield filename filename = outfile.format(start=count, end=count + batchsize) root = start() loc = "{}{}".format(base_url, make_song_url(song)) etree.SubElement(etree.SubElement(root, "url"), "loc").text = loc count += 1 close(root, filename) yield filename

The most important lines in terms of lxml.etree and sitemaps are:

root = etree.Element("urlset", xmlns="http://www.sitemaps.org/schemas/sitemap/0.9") ... etree.SubElement(etree.SubElement(root, "url"), "loc").text = loc

Another important thing is the note about using .values(). If you don't do that Django will create a model instance for every single row it returns of the iterator. That's expensive. See this blog post.

Another important thing is to use a Django ORM iterator as that's much more efficient than messing around with limits and offsets.

Generate the map of sitemaps

Making the map of maps doesn't need to be gzipped since it's going to be tiny.

def generate_map_of_maps(base_url, outfile): root = etree.Element( "sitemapindex", xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" ) with open(outfile, "wb") as f: f.write(b'<?xml version="1.0" encoding="UTF-8"?>\n') files_created = sorted(glob("sitemap-*.xml.gz")) for file_created in files_created: sitemap = etree.SubElement(root, "sitemap") uri = "{}/{}".format(base_url, os.path.basename(file_created)) etree.SubElement(sitemap, "loc").text = uri lastmod = datetime.datetime.fromtimestamp( os.stat(file_created).st_mtime ).strftime("%Y-%m-%d") etree.SubElement(sitemap, "lastmod").text = lastmod f.write(etree.tostring(root, pretty_print=True))

And that sums it up. On my laptop, it takes about 60 seconds to generate 39 of these files (e.g. sitemap-1560000-1600000.xml.gz) and that's good enough.

Bonus and Thoughts

The bad news is that this is about as good as it gets in terms of performance. The good news is that there are no low-hanging fruit fixes. I know, because I tried. I experimented with not using pretty_print=True and I experimented with not writing with gzip.open and instead gzipping the files on later. Nothing made any significant difference. The lxml.etree part of this, in terms of performance, is order of maginitude marginal in comparison to the cost of actually getting the data out of the database plus later writing to disk. I also experimenting with generating the gzip content with zopfli and it didn't make much of a difference.

I originally wrote this code years ago and when I did, I think I knew more about sitemaps. In my implementation I use a batch size of 40,000 so each file is called something like sitemap-40000-80000.xml.gz and weighs about 800KB. Not sure why I chose 40,000 but perhaps not important.

Categories: FLOSS Project Planets

Russ Allbery: podlators 4.12

Planet Debian - Fri, 2019-05-31 23:58

This release only fixes a test suite issue. I've been putting it off for ages because I was hoping to pick up some previous discussions and make some more substantive changes, but that hasn't happened yet and I keep getting mail from failing tests. Worse, a few other people have investigated the problem helpfully, and I don't want to waste more of anyone's time!

Also, I noticed I'd not posted anything but book reviews for this month, so wanted to do at least one software release, even if trivial.

Anyway, sometimes the Encode module gets loaded before the test suite for podlators, which makes it impossible to test the warnings that happen if Encode isn't available. That's fine, except that the test failed entirely in that case, instead of being skipped. This release fixes it to be skipped properly.

You can get the latest release from the podlators distribution page.

Categories: FLOSS Project Planets

Peter Bengtsson: Generate a random IP address in Python

Planet Python - Fri, 2019-05-31 23:10
A fun trick to generate random, but seeded, IPv4 addresses in a Django app.
Categories: FLOSS Project Planets

Paul Wise: FLOSS Activities May 2019

Planet Debian - Fri, 2019-05-31 19:41
Changes Issues Review Administration
  • Debian: answer SSH question, redirect LDAP change mail, fix some critical typos in LDAP, restart bacula after postgres restart
  • Debian wiki: forward questions to page authors, answer wiki support questions, whitelist email domains, whitelist email addresses, ping folks with bouncing email addresses, disable accounts with bouncing email
  • Debian package tracking system: deploy changes, configure automatic disabling of unused oldstable suites
  • Debian derivatives census: restore corrupted file from backups, disable the code that corrupted the file, deploy changes
Communication Sponsors

The leptonlib/tesseract-lang/tesseract/sysstat Debian uploads and the ufw feature request were sponsored by my employer. All other work was done on a volunteer basis.

Categories: FLOSS Project Planets

Mediacurrent: Improve Your Remote Work Productivity With the ‘Shutdown Ritual’

Planet Drupal - Fri, 2019-05-31 15:58

For all its benefits, working remote— as most of our Mediacurrent team does— still has its challenges. 

The one that people ask me about the most is, "how do you keep any kind of work / life balance when your work and home are inseparable?" For me, the answer to this has been what I call my "shutdown ritual." It's basically just how I try to end my workday every day, but I've found putting some thought into a routine has helped a lot to make my evenings more relaxing and my mornings more productive.

In this post, I'm going to cover:

  • what a shutdown ritual is
  • the benefits  of having a shutdown ritual
  • my specific shutdown ritual
What is a shutdown ritual?

So first, what is a shutdown ritual?

A shutdown ritual is a set routine of actions that you perform at the end of each work day to finalize your day and signify that your work day is done.

I got this concept from an excellent book called Deep Work: Rules for Focused Success in a Distracted World, by Cal Newport. 

His core argument in the book is that the most valuable skill in our economy is deep focused work and that is becoming increasingly rare. If you want to set yourself apart, cultivating a deep work ethic is the way to go. He outlines several rules and guidelines you can follow to start cultivating this habit.

One of the tools he recommends is the shutdown ritual.

The bare outline of the shutdown routine that he outlines in Deep Work is:

  1. update all todo lists
  2. read over the todo lists in their entirety (reprioritizing items as necessary)
  3. review the calendar for the next two weeks, make sure any todos required for events are on the todo list
  4. write down a plan for the next day
  5. close everything on your computer
  6. say a magic phrase, like "Shutdown complete" or "I'm outta here"

Newport also discusses the shutdown ritual briefly on his blog: Drastically Reduce Stress with a Work Shutdown Ritual.

Why do a shutdown ritual?

One of Newport's biggest criticisms of modern workers is that we're always on, and because of this, our attention and energy is too dispersed. We can get notifications or just compulsively check Slack or email, even outside of work hours. But deep work requires disciplined attention and energy. If you want to do deep work during the day, you need to make sure you are not constantly doing shallow work (even off the clock). 

Benefit 1: A defined ending

One of the main benefits of the shutdown ritual is having a well-defined end of the workday. Once the shutdown ritual is done, work is done. Don't think about it. Don't worry about it. Don't check email. Don't look at Slack. If you want to be your most productive self, you need to take a complete break until the next work day.

Now it's time to do all the relaxing evening things like feeding your kids and washing the dishes.

When I tell people I work from home, a majority of them say something like, "You must feel like you're working all the time" or "How do you separate your work life from home life?" and I have certainly found it to be a bigger challenge to turn off at the end of the day since I don't really leave "the office." The shutdown ritual helps define the end of the workday and the beginning of being fully present at home.

It has helped me be more productive during the day and helped me have more focused attention with the family after work.

Benefit 2: Confidence in the ending

The important part about the steps in the shutdown routine is giving you confidence that everything you needed to do is done. 

If you're like me, you might be washing the dishes and still be debugging code in your head. Or you might suddenly remember an important email you were supposed to respond to and didn't. Or you might start thinking about a meeting on your schedule tomorrow that you're anxious about.

But the steps of your shutdown ritual should help you capture all these thoughts before you end your workday. You want to capture them and write them down somewhere, so they aren't floating around in your brain all night. 

Sometimes, you might be in the middle of your shutdown ritual and remember something that can't wait until tomorrow. That's fine, go ahead and do it and then start your shutdown ritual over.

Because once your shutdown ritual is over, and a work worry comes into your head, you want to be confident to say to yourself: 

I went through the shutdown ritual. I know that everything important has already been accounted for. Therefore, there is no need to worry.

And move on with your night.

Benefit 3: Having an anchor for other habits

The last benefit I'll mention is that having a shutdown ritual at the end of the day can be a helpful anchor for other habits.

If you have a well-established habit like a shutdown routine, you can leverage it to help establish other habits. James Clear calls this Habit stacking. The idea is that by pairing a new habit with one that already exists, you make it more likely to stick to a new habit.

Some examples of habit stacking with the shutdown ritual:

  • after my shutdown ritual, I will take 5 deep breaths and smile
  • after my shutdown ritual, I will put on my workout clothes and go to the gym
  • after my shutdown ritual, I will call a friend

A shutdown ritual can help you end your workday productively and launch a new habit or hobby to make the rest of your life even better as well!

My Shutdown Ritual

Here's what my shutdown ritual looks like. At 5:15pm everyday, I get a friendly message from slackbot to start my shutdown ritual. 

You can set a reminder in Slack like this:

/remind me to 

Time to start your shutdown ritual!

- reconcile timesheet
- check email for anything requiring urgent response
- add new / outstanding tasks to todo list
- check JIRA
- skim task lists
- check calendar for tomorrow
- make a rough plan for the next day

Shut down complete

every weekday at 5:15pm

For me, this is actually about 45 minutes before the end of my workday. I used to have the reminder for 15 minutes before I signed off, but I could never get to a stopping point and do the shutdown in 15 minutes. 45 minutes gives me enough time to start looking for an exit in my current work and to go through the routine.

Here is what is on my shutdown ritual list:

  • reconcile timesheet
  • check email for anything requiring urgent response
  • check JIRA
  • add new / outstanding tasks to todo list
  • skim task lists
  • check calendar for tomorrow
  • make a rough plan for the next day

Reconcile Timesheets

I take a look at my time tracking software and make sure all my time is accounted for.

Check Email

I go through my email inbox, responding to anything that is really urgent, and adding tasks to Todoist for anything that is not. My goal is to delete as many as possible.

Check JIRA

I open up JIRA to see what tasks are assigned to me, and add them to Todoist, if they aren't already there. I sometimes remember things I said I was going to do but didn't (like update a ticket or assign to someone else) so if I can quickly do that I will.

Skim Todo lists

This gives me a broad view of what's going on. Sometimes I notice a glaring error or remember something that completely slipped my mind, so I add that here. I also sometimes reorder things based on how priorities changed throughout the day.

Check Calendar

Next, I check my calendar for tomorrow. Do I have any meetings tomorrow that I have a deliverable for? Did I complete the deliverable? Any conflicts?

Make a rough plan for the next day

The last item on my list is making a rough plan for the next day. This really sets me up for success in the morning, and helps prevent a slow, groggy start to the work day. Sometimes I make my plan in Todoist just by setting due dates for todos for tomorrow and putting them in the order I want. Other times I have a simple text file where I make a list. It kind of depends on what projects I'm on and the kind of work I'm doing from week to week.

Shut down

I close all applications on my computer. I put the computer to sleep or shut it down. Sometimes I say "shutdown complete"

Sometimes here I tidy up my desk.

I start singing "It's a wonderful day in the neighborhood" and change into my house sweater and house shoes.

The work day is done. I'll be back tomorrow. 

And that's how I end my days productively with the shutdown ritual.

Categories: FLOSS Project Planets

Stack Abuse: The Python String strip() Function

Planet Python - Fri, 2019-05-31 13:49

In this article, we'll examine how to strip characters from both ends of a string in Python.

The built-in String type is an essential Python structure, and comes with a built-in set of methods to simplify working with text data. There are many situations in which a programmer may want to remove unwanted characters, i.e. strip certain characters, from either the beginning or ending of a string.

The most common requirement is to strip whitespace (spaces, tabs, newline characters, etc.) from both ends of a string. This usually occurs after importing raw text data from a file, database, web service, or after accepting user input, which may contain typos in the form of extra spaces. This can be handled by the default usage of the String.strip() method, as seen here:

>>> orig_text = ' The cow jumped over the moon! \n' >>> print(orig_text.strip()) The cow jumped over the moon! >>>

Note that this method does not change the original value of the string, i.e. it does not change the string in-place. It simply returns a new string with the whitespace on either end stripped out. We can verify this by printing out the original string:

>>> print(orig_text) The cow jumped over the moon! >>>

The strip method also enables us to specify which types of characters we want to strip. This can be useful if we want to remove other characters besides whitespace. To do this we simply specify the characters to strip by passing an argument containing these characters to the String.strip() method:

>>> orig_text = '-----The cow jumped over the moon!$$$$$' >>> print(orig_text.strip('-$')) The cow jumped over the moon! >>>

This is useful for removing characters at the start or end of a string that were used for formatting purposes, for example. So if you have a Markdown-formatted string, you can easily remove the header syntax like this:

>>> md_text = '### My Header Here' # Denotes an H3 header in Markdown >>> print(md_text.strip('# ')) My Header Here >>>

Finally, Python provides a way to strip characters from only one side of the string via the String.rstrip() and String.lstrip() methods. These methods work exactly the same way as the String.strip() method, but String.rstrip() only removes characters from the right side of the string and String.lstrip() only removes characters from the left side of the string:

>>> orig_text = '*****The cow jumped over the moon!*****' >>> print(orig_text.rstrip('*')) *****The cow jumped over the moon! >>> print(orig_text.lstrip('*')) The cow jumped over the moon!*****

Once again we can print the original string to see that it was unaffected by these operations:

>>> print(orig_text) *****The cow jumped over the moon!*****


About the Author

This article was written by Jacob Stopak, a software consultant and developer with passion for helping others improve their lives through code. Jacob is the creator of Initial Commit - a site dedicated to helping curious developers learn how their favorite programs are coded. Its featured project helps people learn Git at the code level.

Categories: FLOSS Project Planets

Drupal Association blog: Once again, we are adding a little color for Pride Month

Planet Drupal - Fri, 2019-05-31 13:11

In June last year, we changed our Drupal Association logo on social media platforms to add a little color for Pride Month and I am really happy to say we will be doing the same again this year.

Since last year, we have introduced better tools for understanding the demographic groups in Drupal and I asked for a report on that. Quite rightly, I don’t have access myself - it is very much locked down. I was interested to see that, of those confirmed users who have filled something into the field (including “none” or “prefer not to answer”), 15.8% have indicated they identify as LGBTQIA. That’s far more than I expected!

We would love to continue to improve our statistics; we're asking everyone to complete the data in the demographics field, even if that is to click “none” - it’s a totally valid and useful response. Go to your user profile, click edit, and find it in the “Personal information” tab.

Finally, whilst the Drupal Association is an educational non-profit and does not advocate policy, I have been personally reminded this week of the extraordinary bravery, friendship, and sheer commitment to overcome challenges by people in the LGBTQIA community and I wanted to help celebrate that in any way we can.

You are strong and you are beautiful. Thank you for being you.

Categories: FLOSS Project Planets

Davy Wybiral: Always Secure Your localhost Servers

Planet Python - Fri, 2019-05-31 12:08
Recently I was surprised to learn that web browsers allow any site you visit to make requests to resources on localhost (and that they will happily allow unreported mixed-content). If you'd like to test this out, run an HTTP server on port 8080 (for instance with python -m http.server 8080) and then visit this page.

You should see "Found: HTTP (8080)" listed and that's because the Javascript on that page made an HTTP GET request to your local server to determine that it was running. Chances are it detected other services as well (for instance if you run Tor or Keybase locally).

There are two implications from this that follow:
  1. Website owners could potentially use this to collect information about what popular services are running on your local network.
  2. Malicious actors could use this to exploit vulnerabilities in those services.
Requests made this way are limited in certain ways since they're considered opaque, meaning that the web page isn't able to read the response body or headers. But even with these restrictions a request can do all kinds of nasty things unless the local server is properly secured.

For instance, at one point Redis was vulnerable to these types of attacks because its protocol is text-over-TCP (just like HTTP) so any web page you visit could craft an HTTP POST request that mimics normal Redis commands. There's an excellent write-up on this vulnerability here (which has since been fixed).

Similarly, if you run the popular gaming platform Steam on Linux, at the time of writing this the main application opens a TCP listener on port 57343. I have no idea what that port is for but I do know that visiting this web page while Steam is open will cause the entire UI to freeze, as well as most games, until the tab is closed: [WARNING: don't blame me if it crashes your game] https://wybiral.github.io/steam-block/

This works because the favicon on that page is actually an HTTP GET request to the TCP server which never closes, thus blocking their listener. It may even be possible to attack the server in other ways with specifically crafted requests (I was able to crash the application using a large enough payload).

These types of vulnerabilities are widespread mostly because application developers assume that the server will only be locally-accessible and not available to every website the user visits. Hopefully this is enough proof to justify why better security measures need to be taken with local servers even if you don't intend to forward the ports you're using to the outside world.

So, as developers what can we do to prevent this kind of attack?

For HTTP and WebSocket servers you can add authentication, CSRF protection, and restrict access based on the page origin (which browsers should include in the request headers).

For TCP servers it could be harder depending on your application. You can detect HTTP-like request headers and block the connection (this seems to be what Redis is doing now). Or require some sort of handshake/authentication that the browser won't perform and reject connections based on that.

As far as preventing fingerprinting and service discovery this way... I'm not entirely sure what the best way to prevent this would be other than ungracefully failing the requests (which a lot of HTTP frameworks probably don't make easy). If anyone has any ideas here feel free to drop me a tweet @davywtf.
    Categories: FLOSS Project Planets

    Hook 42: Search and Facets and Queries, Oh My!

    Planet Drupal - Fri, 2019-05-31 11:24
    Search and Facets and Queries, Oh My! Lindsey Gemmill Fri, 05/31/2019 - 15:24
    Categories: FLOSS Project Planets

    Sylvain Beucler: Debian LTS - May 2019

    Planet Debian - Fri, 2019-05-31 11:22

    Here is my transparent report for my work on the Debian Long Term Support (LTS) project, which extends the security support for past Debian releases, as a paid contributor.

    In May, the monthly sponsored hours were split evenly among contributors depending on their max availability - I declared max 30h and got 18h.

    • firefox-esr: jessie-security update, security-ish issue with modules signing authority, backporting stretch's
    • CVE-2018-19969/phpmyadmin: attempt backporting the 49 patches and decide against it since they merely mitigate the CSRF issues but certainly break the testsuite
    • CVE-2018-20839/systemd: attempt to reproduce issue in Jessie, conclude no-dsa due to non-reproducibility and regressions introduced by the patch
    • CVE-2019-2697/openjdk-7: triage (sync with previous uploaders, conclude "not-affected")
    • CVE-2019-0227/axis: triage (clarify SSRF situation, sync with packager, conclude "unfixed")
    • dns-root-data: discuss potential update, conclude not relevent due to no reverse dependencies
    • gradle, kdepim: update triage info

    Incidentally, last month I mentioned how regularly updating a 19MB text file caused issues in Git - it appears it's even breaking salsa.debian.org! Sadly conversation between involved parties appears difficult.

    If you'd like to know more about LTS security, I recommend you check:

    Categories: FLOSS Project Planets

    Sylvain Beucler: Debian LTS - May 2019

    Planet Debian - Fri, 2019-05-31 11:14

    Here is my transparent report for my work on the Debian Long Term Support (LTS) project, which extends the security support for past Debian releases, as a paid contributor.

    In May, the monthly sponsored hours were split evenly among contributors depending on their max availability - I declared max 30h and got 18h.

    • firefox-esr: jessie-security update, security-ish issue with modules signing authority, backporting stretch's
    • CVE-2018-19969/phpmyadmin: attempt backporting the 49 patches and decide against it since they merely mitigate the CSRF issues but certainly break the testsuite
    • CVE-2018-20839/systemd: attempt to reproduce issue in Jessie, conclude no-dsa due to non-reproducibility and regressions introduced by the patch
    • CVE-2019-2697/openjdk-7: triage (sync with previous uploaders, conclude "not-affected")
    • CVE-2019-0227/axis: triage (clarify SSRF situation, sync with packager, conclude "unfixed")
    • dns-root-data: discuss potential update, conclude not relevent due to no reverse dependencies
    • gradle, kdepim: update triage info

    Incidentally, last month I mentioned how regularly updating a 19MB text file caused issues in Git - it appears it's even breaking salsa.debian.org! Sadly conversation between involved parties appears difficult.

    If you'd like to know more about LTS security, I recommend you check:

    Categories: FLOSS Project Planets

    Drudesk: Interesting option for news websites on Drupal — featured news collections

    Planet Drupal - Fri, 2019-05-31 10:46

    Drupal is a great choice for media websites. This is due to easy content editing, flexible moderation workflows, advanced media handling, and much more.

    And, of course, media and news websites on Drupal can enjoy unlimited content display options for. Today, we would like to show you one of them that we implemented for our customer’s Drupal website — so-called featured news collections, aka grouped news.

    Categories: FLOSS Project Planets

    Stack Abuse: The Python zip() Function

    Planet Python - Fri, 2019-05-31 09:55

    In this article, we'll examine how to use the built-in Python zip() function.

    The zip() function is a Python built-in function that allows us to combine corresponding elements from multiple sequences into a single list of tuples. The sequences are the arguments accepted by the zip() function. Any number of sequences can be supplied, but the most common use-case is to combine corresponding elements in two sequences.

    For example, let's say we have the two lists below:

    >>> vehicles = ['unicycle', 'motorcycle', 'plane', 'car', 'truck'] >>> wheels = [1, 2, 3, 4, 18]

    We can use the zip() function to associate elements from these two lists based on their order:

    >>> list(zip(vehicles, wheels)) [('unicycle', 1), ('motorcycle', 2), ('plane', 3), ('car', 4), ('truck', 18)]

    Notice how the output is a sequence of tuples, where each tuple combines elements of the input sequences with corresponding indexes.

    One important thing to note is that if the input sequences are of differing lengths, zip() will only match elements until the end of the shortest list is reached. For example:

    >>> vehicles = ['unicycle', 'motorcycle', 'plane', 'car', 'truck'] >>> wheels = [1, 2, 3] >>> list(zip(vehicles, wheels)) [('unicycle', 1), ('motorcycle', 2), ('plane', 3)]

    Since the wheels list is shorter in this example (3 items as opposed to the 5 that vehicles has), the sequence stopped at "plane".

    As previously mentioned, the zip() function can be used with more than two sequences:

    >>> vehicles = ['unicycle', 'motorcycle', 'plane', 'car', 'truck'] >>> wheels = [1, 2, 3, 4, 18] >>> energy_sources = ['pedal', 'gasoline', 'jet fuel', 'gasoline', 'diesel'] >>> list(zip(vehicles, wheels, energy_sources)) [('unicycle', 1, 'pedal'), ('motorcycle', 2, 'gasoline'), ('plane', 3, 'jet fuel'), ('car', 4, 'gasoline'), ('truck', 18, 'diesel')]

    One reason to connect multiple sequences like this is to create a cleaner way to iterate over the items in multiple sequences. Without the zip() function, we would have to do something like this:

    >>> for i in range(len(vehicles)): ... print('A ' + vehicles[i] + ' has ' + str(wheels[i]) + ' wheels and runs on ' + energy_sources[i]) ... A unicycle has 1 wheels and runs on pedal A motorcycle has 2 wheels and runs on gasoline A plane has 3 wheels and runs on jet fuel A car has 4 wheels and runs on gasoline A truck has 18 wheels and runs on diesel

    But with the zip() function we can use the following cleaner syntax via tuple unpacking:

    >>> for v, w, es in zip(vehicles, wheels, energy_sources): ... print('A ' + v + ' has ' + str(w) + ' wheels and runs on ' + es) ... A unicycle has 1 wheels and runs on pedal A motorcycle has 2 wheels and runs on gasoline A plane has 3 wheels and runs on jet fuel A car has 4 wheels and runs on gasoline A truck has 18 wheels and runs on diesel

    One final thing to understand about the zip() function is that it actually returns an iterator, not a list of tuples. Note that in our first two examples above, we wrapped the zip() function inside the list() type to convert the result to a list. If we tried to display the return value of the zip() function directly we would see something like this:

    >>> zip(vehicles, wheels) <zip object at 0x1032caf48>

    This 'zip object' is an iterable instance of the Zip class, which means it will return its contents one by one in a for-loop, instead of all at once, the way a list does. This is more efficient for large sequences that would be very memory intensive if accessed all at once.

    About the Author

    This article was written by Jacob Stopak, a software consultant and developer with passion for helping others improve their lives through code. Jacob is the creator of Initial Commit - a site dedicated to helping curious developers learn how their favorite programs are coded. Its featured project helps people learn Git at the code level.

    Categories: FLOSS Project Planets

    Mediacurrent: What’s New in Rain 3.0

    Planet Drupal - Fri, 2019-05-31 09:06

    Mediacurrent created the Rain Install Profile to build fast, consistent Drupal websites and improve the editorial experience. Rain expedites website creation, configuration, and deployment.

    The Mediacurrent development team is pleased to announce some new updates to the Rain distribution in version 3.0. We have now made Drupal project template easier to use and maintain by splitting Rain content features (all of which are optional) from the main “base” package. This allows developers flexibility in which features they use while still pre-configuring modules that jump-start development.

    There are some key changes that we will highlight here:

    1. The Rain package has now been split and renamed to mediacurrent/rain and mediacurrent/rain_features respectively. The latter repository now contains all the optional content features and their dependencies while the base package pre-configures the base installation.
    2. An UPDATE doc has been added to the Rain repository which explains in detail how to update from the 2x branch to 3x. This document will be kept up to date with any future changes that require manual changes or explanation.
    3. A few new dependencies have been added while several less frequently used dependencies have been removed. The UPDATE doc gives further details on what was added or removed and how to upgrade.

    Note that the Drupal-project template is only used for provisioning new projects. Any project that currently uses the 2x version of the Rain distribution will not break or be forced to update. Updates to 2x will continue through to Drupal core 8.8 but then be sunset in favor of the 3x branch. Overall the process of updating from 2x to 3x should be relatively painless. 

    If you experience any problems updating please file an issue in the official Rain project queue on Drupal.org: https://www.drupal.org/project/rain.

    Installing Rain 3.0

    To install the Rain distribution, we recommend you leverage our Drupal project template which includes a VM and Guardr security along with the Rain install profile and other tools.

    Our recent article entitled “Drupal 8 Rain & GatsbyJS Integration” covered how to install Rain using the project template so we will recap the first step here which remains the same in version 3.0.

    First you will want to create a repository wherever you typically host your Git projects (i.e. Github, Bitbucket or Gitlab). Once you have that setup you can clone Mediacurrent’s repo and point the origin back to your Git repo.


    git remote set-url origin git@bitbucket.org:mediacurrent/shortcode_project.git

    Next, you will want to initialize the project. You can do that by running the following commands with your local host name and IP.


    composer install

    composer drupal-scaffold

    ./scripts/hobson project:init example.mcdev

    Finally, to build the project and run the install you can simply run “./scripts/build.sh” which runs composer install as well as the Drupal install. Note that the project README has more detailed instructions but this will give you an idea how to get up and running quickly.

    Video Tutorial for Installing Rain

    Categories: FLOSS Project Planets

    OpenSense Labs: Open Source : A community and culture

    Planet Drupal - Fri, 2019-05-31 08:28
    Open Source : A community and culture Harshit Fri, 05/31/2019 - 17:58

    Open source communities are more about sharing ultimate value rather than just building something. They love to contribute and impact people all across the globe. Open source culture is more than just reusing free code on GitHub to get products to market faster.

    The open source culture embraces an approach to software development which totally lays emphasis on all round collaboration and helpful nature, the teams tend to focus more on increased competencies instead of core infrastructure and cross channel implementation.  The culture embraces an approach to software development that emphasizes internal and external collaboration, an increasing focus on core competencies instead of core infrastructure, and implementation of DevOps processes commonly associated with microservices and cloud native technologies. 

    What are the key traits of an open source community and culture? A responsibility to contribute 

    Open source involves a broad range of technologies and a diverse set of people who bring some or the other expertise to the table. Often people are more inclined towards contributing the best of their individual abilities. They feel the responsibility to contribute and make sure they are often involved in the betterment of multiple projects and are they are often people are members of multiple projects, involving a broad range of technologies. Frequently, member recognition isn’t set by how much they’re paid or what titles they’re called. It’s how much of a headache is solved or endured for others. 

     All round responsibility and accountability

    Accountability between members begins when they know each other as people and professionals. It’s especially important to have consistent written contact, ad hoc and scheduled video conferencing, and meeting in person at least once a year to build personal bonds.

    Seamless and undeterred Collaboration

    Collaboration for an open source organization culture stretches across multiple areas. Well into domains like organization goals, cultural fit, and more.

    Team members who define together what a cultural fit is demonstrates what's important to the organization. Just as, a united group of passionate hackers and designers who take part in the joys of community sharing, they want to hire those having like interests and similar ideals.

    More inclination towards automation

    Automating tasks within an organizational culture is about respecting people’s effort while not wanting them to be distracted from getting the right thing done when needed. 

    Organizational members focus ultimately should be on what’s important to them and, in turn, the organization.

    Consistency in everything 

    The consistency of people, processes, and management thereof is the glue of an open source organization culture. Without consistency of action, principles and guidelines flounder despite the best of intentions.

    Streamlined Processes

    It's extremely hard work to develop fundamentals when shortcuts and hacks so often seem to provide great short-term benefits. However, consistency is the key to positive long-term results.  

    For example; there needs to be a detailed and consistent process in hiring for fit, not skills, and for the long-term. Beyond these two key criteria, the candidates should also be demonstrably capable, driven, and passionate for the role to be filled.  

    An underlying passion 

    The underlying spirit to do good work is hard to find, it comes to you upon due search. And it is more powerful than any other driving force in the professional landscape. You will strive harder and harder for the things which mean something to you and the Drupal community makes you feel connected to their growth, you grow as they grow, which ultimately helps you feel the need to deliver sheer excellence. 

    Shared Responsibility

    At an organization, culture becomes the way you work. Through culture, there’s a shared responsibility for good communication and positive results. In communicating with clients and one another, it needs to be timely, considerate, and accurate. 

    Drupal: For great community and culture Drupal has a predefined set of values and principles

    Drupal, since its inception was built around a foundational set of values and principles. The agenda was to gather a community of like minded individuals and bring them on the same page about the vision and mission of the product and its roadmap.

    Source: Dries Buytaert’s Blog Drupal’s code of conduct and CWG 

    Drupal community's Community Working Group comprises of independent volunteers who strive hard to protect and promote the health of the entire Drupal community, they also help and maintain and keep on track the Drupal Code of Conduct and also act as the escalation body to help mediate conflict between community members. 

    What should make you want to contribute to Drupal?

    Can you imagine hundreds of thousands of people relying on your code or waiting to get some feedback from you? Their business’s growth is dependent on the advancements you make in your contribution. The more you contribute the better your worth and stance in the community, what is better than people believing, listening and relying on you for some advancements in the community? 

    Final word 

    Open Source is here to stay and develop software that has a huge impact upon individuals and businesses. People continue to make efforts because of their underlying passion for building great things and open source communities are an example of that. 

    blog banner blog image Open Source work culture Blog Type Articles Is it a good read ? Off
    Categories: FLOSS Project Planets

    Bits from Debian: Debian welcomes its GSoC 2019 and Outreachy interns

    Planet Debian - Fri, 2019-05-31 08:15

    We're excited to announce that Debian has selected seven interns to work with us during the next months: two people for Outreachy, and five for the Google Summer of Code.

    Here is the list of projects and the interns who will work on them:

    Android SDK Tools in Debian

    Package Loomio for Debian

    Debian Cloud Image Finder

    Debian Patch Porting System

    Continuous Integration

    Congratulations and welcome to all the interns!

    The Google Summer of Code and Outreachy programs are possible in Debian thanks to the efforts of Debian developers and contributors that dedicate part of their free time to mentor interns and outreach tasks.

    Join us and help extend Debian! You can follow the interns weekly reports on the debian-outreach mailing-list, chat with us on our IRC channel or on each project's team mailing lists.

    Categories: FLOSS Project Planets

    OpenSense Labs: Myths about OpenSource Technology

    Planet Drupal - Fri, 2019-05-31 08:02
    Myths about OpenSource Technology Harshit Fri, 05/31/2019 - 17:32

    Open source software has been receiving some serious criticism and some serious applauds from the tech community all across the world. People beg to differ on a lot of ideas about it, it has led to some serious publicity over these years. But hold on a second, with publicity comes myth and with myth comes some serious thoughts and people get mislead. Which at the end of the day hampers people’s thought process. Some think source software is totally free to use and some question its security quotient.

    Open source technology has made it nearly possible to do so much in literally no time.

    Let’s get into some myths about open source technology about back them with actual facts.

    Myth #1: Open Source is free

    Many people get more inclined towards open source software because they are misled into thinking that open source software is free to use and there will be hardly any software run costs in the future. On the correct note, this is not true, open source means open source code and that you can easily  access the source code of any system if you are enrolled in any given community.

    To break it into a more understandable concept:

    An Open Source Content Management System vendor can charge you for the services being provided around the open source software, And by far there is no link between the software license and the what you have to pay to get out of it.

    Take Drupal for example, it's free to download and use for personal purposes but the advancements have gone so far that you will need to seek expert services for utilising the software to it's very best potential.

    What's the free part?

    You are free to access the source code behind the functionality and alter it for your own use case but provided that you abide by the terms and conditions in the license agreement.

    Myth #2 - All open source software is Linux based

    This myth is one of the most common and it’s fair enough for people to believe, especially when they are new to the open source landscape or just starting off their careers. When OSS came into the picture, this was the most commonly When people mentioned OSS, the quick and common assumption about it is that OSS only runs on the Linux operating system. It is a quick and easy assumption to make as many open source programs are made with Linux availability as a prime motivator.

    MYTH #3: Contribution to OpenSource is only for startups

    In the government sector, open source contribution is strong and they have the deploy teams to be able to handle and make the best use out of the open source software. Hence they end up making more and more contributions in the process.

    On the other hand, the developers in the public sector do contribute to the code base but they have to expect some benefit out of it, either in the financial aspects or on the career trajectory aspect.  Some state and federal agencies like code.ca.gov and code.gov are using code sharing and collaboration to help the government in cutting down the duplicacy costs.

    Drupal as an open source software is great for giving startups a heads up for showcasing their expertise and content. On the other hand, it is also one of the best solutions when it comes to enterprise requirements. You can custom build your content management systems which serve for a large scale content repository.

    Myth 4 : OSS is less secure than proprietary software

    So, is open source software inherently more secure? Of course not. Before going for any open source solution, you should look into its security thoroughly.

    You can always review its version history and the frequency of security updates provided by the supporting community, you should also look for the amount of work being poured into its security segment and what is the word of mouth like?

    Maybe you’ll even find an independent agency vouching for a product’s security, or certificates proving its reliability, or a respected colleague who can assure you that it's the best option on the market.

    Additionally, you can see what tools your competitors, partners, and established companies in the industry are using. For instance, Ruby on Rails is used by 500px and Airbnb, and that alone is a great indicator that this framework is reliable enough for startups.

    Drupal is considered one of the most secure content management systems across the world. Why? Because of its dedicated security team and the frequent security releases which make the system more and more robust over time.

    Myth 5 - OSS is not scalable

    Open source software is never designed to fit in everyone's shoe, the entire agenda of open source software is to make sure people can make it fit in their shoes with the help of respective expertise and their organisational requirements.

    Take Drupal for example, It is designed to be scalable and adaptable in comparison to its commercial competitors. It is supposed to be evolved by the community and hence meet enterprise expectations. Developers have been able to adapt projects to small  and enterprise size requirements.

    Myth 6 - Open Source is not maintainable

    It is a strong assumption that open source software is harder to maintain and it can lead to possible confusion among the user crowd. There is always a sense of responsibility and motivation to improve the code and better the software overall, not for monetary gain, not for any gain other than a feeling of social responsibility.

    Open Source softwares generally track all the upgrades, improvements and maintenance measures using paid tools to help maintain a record of the versioning and who was the code contributed by. See? The community has already got the maintenance concern or issue covered before it even gets started.

    One more strong foothold about open source is that it can be managed and the work can be overtaken by other technology service providers, in case your technical team decides to move on. So, you should now be sold on the idea that open source is maintained like a premier software.

    Myth 7 - OSS doesn’t have a support system in place

    Since there is no one to hold accountable openly, people think open source software is less cared about or not supported so well in the industry.

    But things are the absolute opposite, the amount of care and support put in by the community support teams is enthralling and it can completely change your mindset about it. Companies which run on the software bring in their brightest minds to help provide support for their software so that they don't get shut down at the end of the day due to lack of sincerity in support and care.

    Final word

    There might be a ton of myths and rumors circulating within and outside the communities but one should always think and work this out before making any harsh assumptions. Myths often keep us from adopting or trying out a technology and this has to come to an end sooner or later because the technology and the community speak for themselves.

    blog banner blog image Open Source Blog Type Articles Is it a good read ? Off
    Categories: FLOSS Project Planets

    OpenSense Labs: Brace yourself, Drupal 9 is Coming!

    Planet Drupal - Fri, 2019-05-31 07:45
    Brace yourself, Drupal 9 is Coming! Jayati Fri, 05/31/2019 - 17:15

    With Drupal 9 on the verge of release in June 2020, the Drupal community has about 18 months to map out a transition plan. The latest versions of Drupal in recent times saw a major breakthrough from the past versions. As the philosophy of development has changed, Drupal 9 is said to be built in Drupal 8 and the migration will be super easy this time.

    Released in 2011, Dries announced the end-of-life (EOL) for Drupal 7 to be due in November 2021 after serving for more than 8 years. However, many people are still on Drupal 7 given the compatibility issues in the two versions which caused major disruption and migration became a task for developers. However, the new philosophy makes it easier to plan and anticipate any unforeseen obstacles that you may encounter. Are you prepared for it?

    Planning for Drupal 9?

    Launching with the objective to modernise the dependencies such as Twigs and Symfony and to remove support of deprecated APIs, Drupal 9 is making its way into the Drupal community soon.

    Every new information being released about the update and new features is gearing us up for the big leap. The first and foremost action to be taken in consideration is to plan and upgrade no later than the summer of 2019. Experts believe, as long as your modules are updated with minor releases like Drupal 8.7 and the upcoming Drupal 8.9 in December 2019, there won’t be much to worry during the main release of Drupal 9. Being upto date with Drupal 8 is a crucial step for adaptability and easier usability in the future.  

    Dries Buytaert wrote recently in a blog:

    ‘’Instead of working on Drupal 9 in a separate codebase, we are building Drupal 9 in Drupal 8. This means that we are adding new functionality as backwards-compatible code and experimental features. Once the code becomes stable, we deprecate any old functionality.’’ What’s New in Drupal 9?

    With a lot of buzz around the new features to be delivered, let’s understand few important reasons for the strategic release of Drupal 9:

    • The innovative model of Drupal 8 had new releases every six months which led to adding of new features and enabling improved ways of problem solving. However, Drupal 9 will deprecate the codes which are needed for backward compatibility. In the process, it will provide an opportunity to remove the codes and anything else that is no longer needed.
    • As of now, Drupal needs to adhere to the vendor support life cycles and integrates with common PHP projects like Twig and Symfony. But the third-party dependencies will reduce with Drupal 9 and we’ll have supported versions of software for a long time such as Twig 2 and Symfony 4/5.
    Why upgrade Drupal 8 when Drupal 9 is coming?

    Drupal 9 is not being built on a new core and its functionalities will not look alien to Drupal 8 users. Instead, they will be added to D8 as backward-compatible code. Only with time and familiarity, as the new features will hold a stable position and mark their success, the older counterparts will be deprecated. As a result, D9 will be stripped of all deprecated code and only the complete collection of stable features will be termed as Drupal 9.

    For example, in Drupal 8.0.0, the Drupal::l($text, $url) was deprecated. Instead of using \Drupal::l(), you can use Link::fromTextAndUrl($text, $url). The \Drupal::l() function was marked for removal as part of some clean-up work.

    What does it Mean…

    With no new paradigms of development and yet being a big leap, how will Drupal 9 change the workings?

    For Core Contributors:

    Your tasks will get limited in Drupal 9 even before the release. Making the quality robust and release more predictable, new features will remove deprecated functionality and lead to Drupal's dependencies to a minimum.    

    For contributed module authors

    Similarly, authors can also start working on the compatibility issues before the release as their Drupal 8 know-how will still remains relevant in Drupal 9 with no dramatic changes in the core.

    For Drupal site owners

    The release of Drupal 9 will make the upgradation much easier for site owners. It will be the same as Drupal 8, only with its deprecated codes removed. According to the experts, keeping your modules and themes stay up-to-date with the latest Drupal 8 APIs will do and a 12- to 18-month upgrade period will be sufficient.

    What happens to module, profile and theme maintainers?

    Though existing Drupal 8 sites have a year and a half to upgrade to Drupal 9, the technology in Drupal 9 would be already battle-tested in Drupal 8. The set of tasks for module and theme maintainers involve getting updated with the new and better APIs. It would be a mandate to check if your code is compatible with Drupal 9 as it may hold invalid when sites migrate. However, do not wait till the release of Drupal 8.8 which is expected at the end of 2019. As six months will be a limited time to upgrade to Drupal 9 for complex codes, it’s advisable to start assessing now.

    How to Prepare for Drupal 9

    The big catch in this whole drill of migration is to make sure that you no longer use the deprecated codes. Following are few ways suggested by Acquia:

    • Be updated with Drupal 8 features and modules
    • Create a report for deprecation using Drupal Check.
    • Check for your active modules which might be deprecated at api.drupal.org
    • Address a consolidated list of errors that can occur and need upgradation to Drupal 9 by generating a ‘’readiness assessment’’.
    • Use the latest versions of dependencies in line with Drupal 9.
    Wrapping it up

    As Drupal 9 will emerge as a phoenix from the ashes of Drupal 8, Buytaert sums it up best, “The big deal about Drupal 9 is that…it should not be a big deal.”

    Excited? Have questions about how Drupal 9 will impact your site? Want to chalk out a plan for upgradation? We are here to help. Drop a line to our experts at hello@opensenselabs.com.

    blog banner blog image Drupal 9 Drupal 8 Drupal Modules Blog Type Tech Is it a good read ? On
    Categories: FLOSS Project Planets

    OpenSense Labs: Run to Glory: The Drupal Effect on High Performance Websites

    Planet Drupal - Fri, 2019-05-31 07:06
    Run to Glory: The Drupal Effect on High Performance Websites Shankar Fri, 05/31/2019 - 16:36

    Usain Bolt, in his last appearance at the World Track and Field Championships in 2017, stood third by a narrow defeat in the 100m race leaving behind a yawning gulf. Bolt finished the race just a hundredth of a second later than his fellow competitors.

    Every (nano)second counts!

    Such is the importance of speed that even a three-time Olympic gold medallist, Usain Bolt, had to bear the brunt of those nanoseconds. Someone might ask “How do I get started learning about web performance?

    Visualise that it is the Mega Book Sale Day and the bookworms are thronging the best performing online stores that are selling the books of renowned authors. Coping with such a colossal turn-up, a site with much faster page load speed would be preferred over the ones that are a bit sluggish. Drupal offers a superb platform for an effective website performance optimisation thereby making it faster and user-friendly.

    The Significance of Website Performance Optimisation

    Web performance optimisation involves monitoring the performance of web application analysing and assessing it, and identifying the best practices to improve it.

    Web applications are a combination of server-side and client-side code. To improve the web performance, both the sides need to be optimised.

    The client-side optimisation relates to the initial page load time, JavaScript that runs in the browser, downloading all of the resources etc. that are seen in the web browser.

    The server-side optimisation relates to database queries and other application dependencies to check how long it takes to run on the server for executing requests.

    Performance optimisation is significant because of the following factors:

    User retention

    BBC found that they are losing out of 10% of users for every extra second their website took to load. Also, DoubleClick by Google found that if the web page took more than 3 seconds to load, 53% of mobile site visitors tend to abandon the page.


    We all strive to make our users engage in a meaningful interaction with what we have built for the web.

    So, if it is an online store, you would like to see a prospective audience turning into buyers. Or if it is a social networking web application, you would want your online visitors to get ensconced in an arresting interaction with one another. High performing sites play a vital role in engaging and retaining users.

    An increase in user retention by 5% can result in increased profits by up to 95%.

    It costs 5 to 25 times more to attract new customers. So, even a 5% enhancement in customer retention can lead to increased profits of 25%-95%.

    By redesigning their web pages, Pinterest combated a 40% reduction in perceived wait times and witnessed a 15% increase in their search engine traffic and sign-ups.

    COOK, a provider of high-quality frozen meals, was able to address the average page load time and cut it down by 850 milliseconds which resulted in 7% in conversions, 10% increase in pages per session and 7% decrease in bounce rate.

    Improved Conversions

    User retention ultimately leads to better conversion rates. Slow sites can have huge repercussions on the business revenues. Better performance of sites can be highly profitable to shore up revenues.

    Source: Hubspot

    According to 2016 Q2 Mobile Insights Report by Mobify, 1.11% increase in session-based conversion was seen for every 100ms decrease in homepage load speed. Moreover, a 1.55% increase in session-based conversion was noticed for every 100ms decrease in checkout page load time. The outcome was an increase in the average annual revenue by approximately $530,000.

    Also, AutoAnything revved up their sales by 12-13% after decreasing their page load time by half.

    User experience

    When sites ship tons of code, underwhelming performance persists as the browsers chew through megabytes of it on snail-paced networks. 

    Source: Impactbnd

    Even the devices with limited processing power and memory can find it hard to cope up with the modest amount of unoptimised code. With poor performance taking centre stage, application responsiveness and availability diminishes.

    Better optimised code lead to high functioning and better-performing sites which in return alleviate the digital user experience.

    Strategising the web performance

    Formulation of strategies to improve web performance can be done in two ways:

    Bottom-up strategy

    Also known as performance-by-design, the bottom-up strategy is the preferred approach to integrate performance as a core development principle. In this strategy, the performance optimisation principles are framed, applied and maintained. This is done right from the application design phase. 

    The key stages that are involved in this approach are stated below:

    • Performance principles are laid out.
    • The key pages/transactions are identified, optimised accordingly, and then performance principles are executed.
    • Performance SLAs (Service Level Agreement) are monitored and maintained.

    Here's a chart by Infosys which explains it best: 

    Key stages involved in bottom-up strategyTop-down strategy

    If an existing application needs to be optimised for performance, top-down strategy comes into play. This is a preferred option only when the legacy applications are being optimised for high performance. Also, this is not cost effective and the optimisation options are limited.

    Steps involved in this strategy are as follows:

    1. Factors that are contributing to the page performance are assessed using tools like PageSpeed Insights, WebPageTest etc.
    2. Activities that would lead to maximum performance improvements are optimised.
    3. Other optimisations with subsequent releases are iteratively implemented.

    In addition to these strategies, one must consider an important methodology called ‘Performance Budgeting’. It means setting a performance threshold that you aim to stay within. You can safeguard your site speed and detect any regression in the performance by setting up a performance budget to ensure continual eye on performance.

    This is how we do it!

    Expected load time and Google page speed score, as shown below, is the core of our perpetual and iterative development process.

    The above chart shows that, while applying performance budgeting methodology, we take note of:

    1. Average load time of 2 seconds or less
    2. Defined maximum limit on page size and number of HTTP requests
    3. Verification of all server site tuning for an efficient and responsive site
    4. Google page speed performance grade of above 90
    5. Implementing performance optimisation
    Implementing Performance Optimisation

    How to speed up my Drupal website performance? Drupal is loaded with an enormous amount of features which, when implemented smartly, can lead to superfast page loads. There are several techniques to make your website faster by leveraging the amazing features of Drupal.

    Keeping your site and modules updated

    Outmoded modules can deter your efforts in speeding up your website. Thus, it is important to update every module enabled on your Drupal site.

    Uninstalling unused modules

    Like those outdated modules, it is significant to keep a tab on least used or no longer used modules. The number of Drupal modules installed on the site is directly proportional to the time taken for code execution which affects page load time. Uninstalling unwanted modules can alleviate execution time.

    Moreover disabling the modules also adds to the execution time of the code. So, a complete removal by uninstalling the unused modules can speed up the Drupal site.

    Optimising Cache

    Optimisation of native cache system ensures that all the web page components are stored in an easily accessible location after a user visits your site for the very time. So, whenever the user visits your site again, the page elements are loaded from the cache which leads to increased page load speed.

    Drupal has the provision of advanced caching with a great set of modules:

    • Internal Page Cache module helps in caching the web pages for anonymous users to increase the speed for subsequent users.
    • Dynamic Page Cache module caches web pages for the anonymous and authenticated users and is recommended for the websites of all screen sizes.
    • BigPipe module allows your users to quickly see the unchanged, cacheable page elements while the personalised content is exhibited next. This technology was inspired by Facebook. Drupal 8’s much improved render pipeline and render API is of huge help.
    • Redis module helps in integrating Drupal with Redis key-value store thereby providing a robust cache system for static pages.
    • Varnish module lets you integrate Drupal sites with an advanced and fast reverse-proxy system - Varnish cache -  to serve static files and unknown page-views quicker and at high volumes.
    Optimising database

    Website coding is not the sole thing that can be optimised. Optimising database by regularly cleaning up the data and removing the unwanted piece of information.

    Memcache API and Integration module, help in the integration of Drupal and Memcached. It stores your data in active memory for a limited period of time thereby making it faster to access. 

    So, instead of making queries to the database constantly, the information is readily available. Such a system also works on the shared web hosting plans.

    Incorporating a Content Delivery Network (CDN)

    Components like CSS, JavaScript and media are hosted by CDN and served to the online visitors from the nearest location. This can help in mitigating the page load time by rapidly delivering web page components.

    Drupal module, CDN, helps in the integration of Content Delivery Network for Drupal websites. It changes the file URLs so that files like CSS, JavaScripts, images, videos, and fonts are downloaded from the CDN instead of your web server.

    Optimising bandwidth

    Aggregating all CSS and JavaScript files to make them load together is what bandwidth optimisation refers to. Such a parallel processing ensures that all the page elements can be seen by the users almost immediately.

    Optimising images

    Drupal 8 core is loaded with image optimisation feature to set the compression ratio of the images and fine-tune the page performance.

    Moreover, the size of the images for screen sizes of different devices can be optimised in Drupal 8 to enhance the page load speed.

    Handling 404 errors

    Whenever something on the website breaks to cause a 404 error, it can lead to sluggishness. For instance, a failed image can damage the performance of the site. Drupal 8 provides a module called Fast 404 which utilises the resources better and whitelists files and verifies pathways of problem.

    Managing the use of CSS and JavaScript

    CSS and JavaScript provide wonderful methods for customisation and flexibility. But, too much of good things can be troublesome for your websites. Avoiding excessive use of CSS files and JavaScript use and keeping the code to a minimum can improve performance.

    Advanced CSS/JS Aggregation, Drupal module, can help in keeping a tab of your front-end performance by aggregating CSS and JavaScript files to improve speed.

    Using lazy loading

    Lazy or on-demand loading is a perfect way to optimise your site’s performance. In this method, you split your code at logical breakpoints and then load it once the user has done something that requires a new block of code.

    Basically, in traditional websites, all the images and content are preloaded into the web browser when someone accesses the site. Lazy loading loads these elements as soon as a user scrolls to view a content.

    Blazy, Drupal module, provides the functionalities of lazy loading and multi-serving the images to save bandwidth and server requests.

    Better web hosting

    It is of consummate importance that, while implementing every possible tips and trick and utilising the Drupal’s amazing features, you chose the best web hosting provider that will decide your site’s ultimate speed, stability and security.

    Upgrading the server hardware

    Server scaling is of paramount importance in order to optimise the website. And to do so, you can either upgrade the server hardware by scaling vertically or by scaling horizontally. When you scale vertically, more resources are thrown at the same server and is considered the simplest approach of scaling the hardware. And when you scale horizontally, more servers are added to separate the load. This approach, when executed well, can minimise the load that any single server receives. In case, you have multiple app servers for Drupal, you will need a method of deploying code to each server concurrently. For example, plartform.sh and pantheon.io can manage the entire hosting setup for you but if you are handling it by yourself, you would require rsync setup or git push to each of your servers etc.

    Case Study

    The Drupal website of the Farm Journal’s MILK was optimised for high performance and better search engine rankings with a help of carefully drafted audit report by Opensense Labs.

    In this section, we will focus on how we used our Drupal expertise to resolve the performance issues.

    Project highlights

    Previously segregated CSS and JS files cached separately which escalated the page load time. We aggregated all these files and put them in one place which assuaged the page load time.

    Moreover, we used Advanced CSS/JS Aggregation Drupal module to minify CSS, JS and HTML and reduce load time.

    In addition to these, we enabled Redis, used as a database, cache and message broker, so that it can be used as the backend instead of MySQL. This allowed cached items to be retrieved swiftly and improved performance.

    Project outcome

    On testing the performance metrics on tools like PageSpeed Insights and Pingdom, we witnessed significant improvement.

    PageSpeed Insights

    • Result on handheld devices
    Pre-implementation (Live Instance)


    Post-implementation (Live Instance)


    • Result on Desktop
    Pre-implementation (Live Instance)


    Post-implementation (Live Instance)



    Pre-implementation Pingdom Score (Live Environment)


    Post-implementation Pingdom Score (Live Environment)



    Speed can be the determining factor in the amount of time an online user spends on your website. It’s important that you remove the sluggishness from your website and inculcate betterments in its performance. Drupal 8 can help by incorporating wonderful features to make your site a high performing space.

    Feel free to reach us at hello@opensenselabs.com for developing a high performing Drupal website

    blog banner blog image Performance Optimisation Web Performance Performance Budgeting Website Performance Optimisation User Retention Conversion Rate User experience Page Load Speed Page Load time Blog Type Articles Is it a good read ? On
    Categories: FLOSS Project Planets