FLOSS Project Planets

Python Data: Python and AWS Lambda – A match made in heaven

Planet Python - Tue, 2017-09-19 09:42

In recent months, I’ve begun moving some of my analytics functions to the cloud. Specifically, I’ve been moving them many of my python scripts and API’s to AWS’ Lambda platform using the Zappa framework.  In this post, I’ll share some basic information about Python and AWS Lambda…hopefully it will get everyone out there thinking about new ways to use platforms like Lambda.

Before we dive into an example of what I’m moving to Lambda, let’s spend some time talking about Lambda. When I first heard about, I was a confused…but once I ‘got’ it, I saw the value. Here’s the description of Lambda from AWS’ website:

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service – all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

Once I realized how easy it is to move code to lambda to use whenever/wherever I needed it, I jumped at the opportunity.  But…it took a while to get a good workflow in place to simplify deploying to lambda. I stumbled across Zappa and couldn’t be happier…it makes deploying to lambda simple (very simple).

OK.  So. Why would you want to move your code to Lambda?

Lots of reasons. Here’s a few:

  • Rather than host your own server to handle some API endpoints — move to Lambda
  • Rather than build out a complex development environment to support your complex system, move some of that complexity to Lambda and make a call to an API endpoint.
  • If you travel and want to downsize your travel laptop but still need to access your python data analytics stack move the stack to Lambda.
  • If you have a script that you run very irregularly and don’t want to pay $5 a month at Digital Ocean — move it to Lambda.

There are many other more sophisticated reasons of course, but these’ll do for now.

Let’s get started looking at python and AWS Lambda.  You’ll need an AWS account for this.

First – I’m going to talk a bit about building an API endpoint using Flask. You don’t have to use flask, but its an easy framework to use and you can quickly build an API endpoint with it with very little fuss.  With this example, I’m going to use Lambda to host an API endpoint that uses the Newspaper library to scrape a website, pull down the text and return that text to my local script.

Writing your first Flask + Lambda API

To get started, install Flask,Flask-Restful and Zappa.  You’ll want to do this in a fresh environment using virtualenv (see my previous posts about virtualenv and vagrant) because we’ll be moving this up to Lambda using Zappa.

pip install flask flask_restful zappa

Our flask driven API is going to be extremely simple and exist in less than 20 lines of code:

from flask import Flask from newspaper import Article from flask_restful import Resource, Api app = Flask(__name__) api = Api(app) class hello(Resource): def get(self): return "Hello World" api.add_resource(hello, '/hello') if __name__ == '__main__': app.run(debug=True, host='0.0.0.0', port=5001)

Note: The ‘host = 0.0.0.0’ and ‘port=50001’ are extranous and are how I use Flask with vagrant. If you keep this in and run it locally, you’d need to visit http://0.0.0.0:5001 to view your app.

The last thing you need to do is build your requirements.txt file for Zappa to use when building your application files to send to Lambda. For a quick/dirty requirements file, I used the following:

zappa newspaper flask flask_restful

Now…let’s get this up to lambda.  With zappa, its as easy as a couple of command line instructions.

First, run the init command from the command line in your virtualenv:

zappa init

You should see something similar to this:

You’ll be asked a few questions, you can hit ‘enter’ to take the defaults or enter your own. For this eample, I used ‘dev’ for the environment name (you can set up multiple environments for dev, staging, production, etc) and made a S3 bucket for use with this application.

Zappa should realize you are working with Flask app and automatically set things up for you. It will ask you what the name of your Flask app’s main function is (in this case it is api.app). Lastly, Zappa will ask if you want to deploy to all AWS regions…I chose not to for this example. Once complete, you’ll have a zappa_settings.json file in your directory that will look something like the following:

{ "dev": { "app_function": "api.app", "profile_name": "default", "s3_bucket": "DEV_BUCKET_NAME" #I removed the S3 bucket name for security purposes } }

I’ve found that I need to add more information to this json file before I can successfully deploy. For some reason, Zappa doesn’t add the “region” to the settings file. I also like to add the “runtime” as well. Edit your json file to read (feel free to use whatever region you want):

{ "dev": { "app_function": "api.app", "profile_name": "default", "s3_bucket": "DEV_BUCKET_NAME", "runtime": "python2.7", "aws_region": "us-east-1" } }

Now…you are ready to deploy. You can do that with the following command:

zappa deploy dev

Zappa will set up all the necessary configurations and systems on AWS AND zip up your libraries and code and push it to Lambda.   I’ve not found another framework as easy to use as Zappa when it comes to deploying…if you know of one feel free to leave a comment.

After a minute or two, you should see a “Deployment Complete: …” message that includes the endpoint for your new API. In this case, Zappa built the following endpoint for me:

https://4wq2muonbb.execute-api.us-east-1.amazonaws.com/dev

If you make some changes to your code and need to update Lambda, Zappa makes it easy to do that with the following command:

zappa update dev

Additionally, if you want to add a ‘production’ lambda environment, all you need to do is add that new environment to your settings json file and deploy it. For this example, our settings file would change to:

{ "dev": { "app_function": "api.app", "profile_name": "default", "s3_bucket": "DEV_BUCKET_NAME", "runtime": "python2.7", "aws_region": "us-east-1" }. "prod": { "app_function": "api.app", "profile_name": "default", "s3_bucket": "PROD_BUCKET_NAME", "runtime": "python2.7", "aws_region": "us-east-1" } }

Next, do a deploy prod and your production environment is ready to go at a new endpoint.

zappa deploy prod

Interfacing with the API

Our code is pushed to Lambda and ready to start accepting requests.  In this example’s case, all we are doing is returning “hello world” but you can see the power in this for other functionality.  To check out the results, just open a browser and enter your Zappa Deployment URL and append /hello to the end of it like this:

https://4wq2muonbb.execute-api.us-east-1.amazonaws.com/dev/hello

You should see the standard “Hello World” response in your browser window.

You can find the code for the lambda api.py function here.

Note: At some point, I’ll pull this endpoint down…but will leave it up for a bit for users to play around with.

 

The post Python and AWS Lambda – A match made in heaven appeared first on Python Data.

Categories: FLOSS Project Planets

Annertech: 5 Reasons to Stop Using Static Design Tools and Start Designing in the Browser

Planet Drupal - Tue, 2017-09-19 08:32
5 Reasons to Stop Using Static Design Tools and Start Designing in the Browser

I'll be presenting at DrupalCon Vienna next week as part of my evangelising against static design tools like Photoshop, InVision, and Sketch. The talk will cover items such as "What's the problem we are trying to solve?", "Why do static tools not solve the problem?", and "Why is working with component design and design in the browser the most sustainable solution?".

I got a request today from a former colleague:

Categories: FLOSS Project Planets

Agiledrop.com Blog: AGILEDROP: Agiledrop going to DrupalCon Vienna!

Planet Drupal - Tue, 2017-09-19 05:15
There have been many blog post written about the forthcoming DrupalCon in Vienna. Many topics were covered including our Accepted Business sessions for DrupalCon Vienna. To refresh your memories, we presented them because our commercial director Iztok Smolic was selected in a business track team to help prepare the program and select the sessions. Maybe it is obvious or maybe it is not. But it's definitely time to say that we will be present on a DrupalCon in Vienna! Besides Iztok, who will be attending his eight DrupalCon, with the first one dating back to 2009, our development director… READ MORE
Categories: FLOSS Project Planets

Python Piedmont Triad User Group: PYPTUG monthly meeting: Plotly, dash and company

Planet Python - Tue, 2017-09-19 04:48
Come join PYPTUG at out next monthly meeting (September 19th 2017) to learn more about the Python programming language, modules and tools. Python is the ideal language to learn if you've never programmed before, and at the other end, it is also a tool that no expert would do without. Monthly meetings are in addition to our project nights.

WhatMeeting will start at 6:00pm.
Main Talk: "Plotly, dash and company"

by Francois Dion
Remake of W. Playfair's classic visualization (source: Plot.ly)
Abstract:There are many visualization packages available out there, each best suited to specific scenarios. In the past several years, I've covered Matplotlib, Seaborn, Vincent, ggplot, 3d visualizations through matplotlib, D3.js and mpld3 and Bokeh. In this presentation we will cover plotly (for javascript, R and Python) and related packages and when it makes sense to use it.
Bio:
Francois Dion is the founder and Chief Data Scientist of Dion Research LLC, specializing in analytics, data science, IoT and visualization. 
He is the author of several open source software, such as stemgraphic (www.stemgraphic.org), the founder of the Python user group for the Piedmont Triad of North Carolina (www.pyptug.org) and mentors various groups in Python, R and analytics at large. You might have run across his multiple part series on LinkedIn on data science books, including part V on Visualization.
When:
Please note, this meeting will be one week early in the month compared to our normal schedule:
Tuesday, September 19th 2017Meeting starts at 6:00PM
Where:Wake Forest University, close to Polo Rd and University Parkway:
Manchester Hallroom: Manchester 241 Wake Forest University, Winston-Salem, NC 27109
 Map this
See also this campus map (PDF) and also the Parking Map (PDF) (Manchester hall is #20A on the parking map)
And speaking of parking:  Parking after 5pm is on a first-come, first-serve basis.  The official parking policy is:"Visitors can park in any general parking lot on campus. Visitors should avoid reserved spaces, faculty/staff lots, fire lanes or other restricted area on campus. Frequent visitors should contact Parking and Transportation to register for a parking permit."Mailing List:Don't forget to sign up to our user group mailing list:
https://groups.google.com/d/forum/pyptug?hl=en
It is the only step required to become a PYPTUG member.
Please RSVP so we have enough food for people attending!RSVP on meetup:https://www.meetup.com/PYthon-Piedmont-Triad-User-Group-PYPTUG/events/242721091/
Categories: FLOSS Project Planets

S. Lott: Three Unsolvable Problems in Computing

Planet Python - Tue, 2017-09-19 04:07
The three unsolvable problems in computing:

  • Naming
  • Distributed Cache Coherence
  • Off-By-One Errors

Let's talk about naming.

The project team decided to call the server component "FlaskAPI".

Seriously.

It serves information about two kinds of resources: images and running instances of images. (Yes, it's a kind of kubernetes/dockyard lite that gives us a lot of control over servers with multiple containers.)

The feature set is growing rapidly. The legacy name needs to change. As we move forward, we'll be adding more microservices. Unless they have a name that reflects the resource(s) being managed, this is rapidly going to become utterly untenable.

Indeed, the name chosen may already be untenable: the name doesn't reflect the resource, it reflects an implementation choice that is true of all the microservices. (It's a wonder they didn't call it "PythonFlaskAPI".)

See https://blogs.mulesoft.com/dev/api-dev/best-practices-for-building-apis/ for some general guidelines on API design.

These guidelines don't seem to address naming in any depth. There are a few blog posts on this, but there seem to be two extremes.

  • Details/details/details. Long paths: class-of-service/service/version-of-service/resources/resource-id kind of paths. Yes. I get it. The initial portion of the path can then route the request for us. But it requires a front-end request broker or orchestration layer to farm out the work. I'm not enamored of the version information in the path because the path isn't an ontology of the entities; it becomes something more and reveals implementation details. The orchestration is pushed down the client. Yuck.
  • Resources/resource. I kind of like this. The versioning information can be in the Content-Type header: application/json+vnd.yournamehere.vx+json.  I like this because the paths don't change. Only the vx in the header. But how does the client select the latest version of the service if it doesn't go in the path? Ugh. Problem not solved.
I'm not a fan of an orchestration layer. But there's this: https://medium.com/capital-one-developers/microservices-when-to-react-vs-orchestrate-c6b18308a14c  tl;dr: Orchestration is essentially unavoidable.
There are articles on choreography. https://specify.io/concepts/microservices the idea is that an event queue is used to choreograph among microservices. This flips orchestration around a little bit by having a more peer-to-peer relationship among services. It replaces complex orchestration with a message queue, reducing the complexity of the code.
On the one hand, orchestration is simple. The orchestrator uses the resource class and content-type version information to find the right server. It's not a lot of code.
On the other hand, orchestration is overhead. Each request passes through two services to get something done. The pace of change is slow. HATEOAS suggests that a "configuration" or "service discovery" service (with etags to support caching and warning of out-of-date cache) might be a better choice. Clients can make a configuration request, and if cache is still valid, it can then make the real working request.
The client-side overhead is a burden that is -- perhaps -- a bad idea. It has the potential to make  the clients very complex. It can work if we're going to provide a sophisticated client library. It can't work if we're expecting developers to make RESTful API requests to get useful results. Who wants to make the extra meta-request all the time?
Categories: FLOSS Project Planets

Talk Python to Me: #130 10 books Python developers should be reading

Planet Python - Tue, 2017-09-19 04:00
One of the hallmarks of successful developers is continuous learning. The best developers I know don't just keep learning, it's one of the things that drives them. That's why I'm excited to bring you this episode on 10 books Python developers should read.
Categories: FLOSS Project Planets

ADCI Solutions: Drupal Global Training Day #5

Planet Drupal - Tue, 2017-09-19 03:12

We used to feel satisfied with just giving a few lectures at Drupal Global Training Day. What do we see now? We see young Drupal developers who are hungry not only for knowledge but for practice, too.

 

Learn about the easy way of organizing a practical part at GTD.

Categories: FLOSS Project Planets

Catalin George Festila: The numba python module - part 002 .

Planet Python - Tue, 2017-09-19 01:48
Today I tested how fast is jit from numba python and fibonacci math function.
You will see strange output I got for some values.
First example:
import numba
from numba import jit
from timeit import default_timer as timer

def fibonacci(n):
a, b = 1, 1
for i in range(n):
a, b = a+b, a
return a
fibonacci_jit = jit(fibonacci)

start = timer()
fibonacci(100)
duration = timer() - start

startnext = timer()
fibonacci_jit(100)
durationnext = timer() - startnext

print(duration, durationnext)The result of this run is:
C:\Python27>python numba_test_003.py
(0.00018731270733896962, 0.167499256682878)

C:\Python27>python numba_test_003.py
(1.6357787798437412e-05, 0.1683614083221368)

C:\Python27>python numba_test_003.py
(2.245186560569841e-05, 0.1758382003097716)

C:\Python27>python numba_test_003.py
(2.3093347480146938e-05, 0.16714964906130353)

C:\Python27>python numba_test_003.py
(1.5395564986764625e-05, 0.17471143739730277)

C:\Python27>python numba_test_003.py
(1.5074824049540363e-05, 0.1847134227837042)As you can see the fibonacci function is not very fast.
The jit - just-in-time compile is very fast.
Let's see if the python source code may slow down.
Let's see the new source code with jit will not work well:
import numba
from numba import jit
from timeit import default_timer as timer

def fibonacci(n):
a, b = 1, 1
for i in range(n):
a, b = a+b, a
return a
fibonacci_jit = jit(fibonacci)

start = timer()
print fibonacci(100)
duration = timer() - start

startnext = timer()
print fibonacci_jit(100)
durationnext = timer() - startnext

print(duration, durationnext)The result is this:
C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0002334994022992635, 0.17628787910376)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0006886307922204926, 0.17579169287387408)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0008105123483657127, 0.18209553525407973)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.00025466830415606486, 0.17186550306131188)

C:\Python27>python numba_test_003.py
927372692193078999176
1445263496
(0.0007348174871807866, 0.17523103771560608)
The result for value 100 is not the same: 927372692193078999176 and 1445263496.
First problem is:
The problem is that numba can't intuit the type of lookup. If you put a print nb.typeof(lookup) in your method, you'll see that numba is treating it as an object, which is slow.
The second problem is the output but can be from same reason.
I test with value 5 and the result is :
C:\Python27>python numba_test_003.py
13
13
13
13
(0.0007258367409385072, 0.17057997338491704)

C:\Python27>python numba_test_003.py
13
13
(0.00033709872502270044, 0.17213235952108247)

C:\Python27>python numba_test_003.py
13
13
(0.0004836773333341886, 0.17184433415945508)

C:\Python27>python numba_test_003.py
13
13
(0.0006854233828482501, 0.17381272129120037)
Categories: FLOSS Project Planets

Catalin George Festila: The numba python module - part 001 .

Planet Python - Tue, 2017-09-19 01:17
Today I tested the numba python module.
This python module allow us to speed up applications with high performance functions written directly in Python.
The numba python module works by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically.
The code can be just-in-time compiled to native machine instructions, similar in performance to C, C++ and Fortran.
For the installation I used pip tool:
C:\Python27>cd Scripts

C:\Python27\Scripts>pip install numba
Collecting numba
Downloading numba-0.35.0-cp27-cp27m-win32.whl (1.4MB)
100% |################################| 1.4MB 497kB/s
...
Installing collected packages: singledispatch, funcsigs, llvmlite, numba
Successfully installed funcsigs-1.0.2 llvmlite-0.20.0 numba-0.35.0 singledispatch-3.4.0.3

C:\Python27\Scripts>pip install numpy
Requirement already satisfied: numpy in c:\python27\lib\site-packages
The example test from official website working well:
The example source code is:
from numba import jit
from numpy import arange

# jit decorator tells Numba to compile this function.
# The argument types will be inferred by Numba when function is called.
@jit
def sum2d(arr):
M, N = arr.shape
result = 0.0
for i in range(M):
for j in range(N):
result += arr[i,j]
return result

a = arange(9).reshape(3,3)
print(sum2d(a))The result of this run python script is:
C:\Python27>python.exe numba_test_001.py
36.0Another example using just-in-time compile is used with Numba’s jit function:
import numba
from numba import jit

def fibonacci(n):
a, b = 1, 1
for i in range(n):
a, b = a+b, a
return a

print fibonacci(10)

fibonacci_jit = jit(fibonacci)
print fibonacci_jit(14)Also you can use jit is as a decorator:
@jit
def fibonacci_jit(n):
a, b = 1, 1
for i in range(n):
a, b = a+b, a

return aNumba is a complex python module because use compiling.
First, compiling takes time, but will work specially for small functions.
The Numba python module tries to do its best by caching compilation as much as possible though.
Another note: not all code is compiled equal.
Categories: FLOSS Project Planets

Catalin George Festila: The beauty of Python: subprocess module - part 004 .

Planet Python - Tue, 2017-09-19 01:17
This series of python tutorials that we started at the beginning of this blog and called "The beauty of Python" is part of the series of tutorials aimed at the simplicity and beauty of the python programming language.
The main goal for us is how to use this programming language in everyday life with different tasks.
Today I will come up with examples to cover this goal and show you how to use the subprocess python module.
  • using the powershell with python :
  • >>> import subprocess
    >>> process=subprocess.Popen(["powershell","Get-Childitem C:\\Windows\\*.log"],stdout=subprocess.PIPE);
    >>> result=process.communicate()[0]
    >>> print result
  • get and print the hostname :
  • >>> print subprocess.check_output("hostname")
  • print the output of ping command :
  • >>> print subprocess.check_output("ping localhost", shell=True)
  • print the output of dir command :
  • >>> cmd = 'dir *'
    >>> supcmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    >>> print supcmd.communicate()[0]
  • run the python script like python shell :
  • >>> import sys
    >>> import subprocess
    >>> pid = subprocess.Popen([sys.executable, "calc.py"])
Categories: FLOSS Project Planets

Code Positive: Automate Drupal 8 Deployment On Pantheon With Quicksilver Scripts

Planet Drupal - Mon, 2017-09-18 21:50

Drupal deployment automation for Pantheon hosting - save time, make deployments safer, automate your testing workflow, and leave the boring repetative work to the computers!

READ MORE

 

Categories: FLOSS Project Planets

Yasoob Khalid: 13 Python libraries to keep you busy

Planet Python - Mon, 2017-09-18 21:22

Hi guys! I was recently contacted by folks from AppDynamics (a part of CISCO). They shared an infographic with me which listed 13 Python libraries. These libraries were categorized in sections. I loved going through that infographic. I hope you guys will enjoy it too. 

Source: AppDynamics


Categories: FLOSS Project Planets

OSTraining: Dropdown Menus in Drupal 8 with the Superfish Module

Planet Drupal - Mon, 2017-09-18 21:00

If you want to build a large, multi-level drop-down menu in Drupal 8, then the Superfish module is a great choice.

The Superfish module makes use of the jQuery Superfish menu plugin, which is useful for multi-level drop-down menus. Superfish has more features than most dropdown menus. It supports touch devices and keyboard interaction.

Categories: FLOSS Project Planets

Daniel Bader: Contributing to Python Open-Source Projects

Planet Python - Mon, 2017-09-18 20:00
Contributing to Python Open-Source Projects

How can you become a contributor on popular, “high-profile” Python open-source projects like Django, Requests, and so on?

Contributing to open-source projects is a great way to build your programming skills, take part in the community, and to make a real impact with your code…

It can also help you get a job as a professional Python developer, but becoming a contributor in the first place—that’s often tough.

So, let’s talk about this question I got from newsletter member Sudhanshu the other day:

Hi Dan,

I am student from India, I don’t really know whether this is your field or not but I have been doing Django development for 5 to 6 months.

I have made few projects on REST APIs, websites, etc. Then I decided to contribute in Django open source projects, particularly those by the Django organization and Mozilla.

What should I do at this point? How can I improve my level of Python knowledge so that I can contribute to these projects?

It sounds like Sudhanshu is in a good spot already.

I love the fact that he’s been working on his own side-projects to build up a portfolio—that’ll be a great asset when he goes job hunting.

If you’re in Sudhanshu’s shoes right now, here’s what I’d focus on next:

Try to strike up some personal connections with people working on those “high-profile” Python projects you want to contribute to.

See if you can make contact somehow—are they on Twitter? Can you comment or ask a question on a GitHub issue? Maybe you can even cold-email them…

Little by little, you’ll be able to build relationships with some of them. Building trust takes a lot of time and dedication, but eventually the timing will be right to offer your help:

Just ask them if there’s something small you could contribute to, like cleaning up the documentation, or fixing typos—simple things like that.

Open-source maintainers usually appreciate it when others help improve the documentation of a project. So that’s often a good way for you to get the foot in the door, metaphorically speaking.

What I want to say is this:

Getting your contributions accepted comes down much more to having built trust with the right people, rather than “throwing a bunch of code over the wall” and creating random pull-requests.

If you’re interested in some more thoughts on this topic, check out the YouTube video I recorded. It contains additional tips and tactics that will help you break into the open-source world:

» Subscribe to the dbader.org YouTube Channel for more Python tutorials.

Good luck on your Python open-source journey and…Happy Pythoning!

Categories: FLOSS Project Planets

Plasma Mobile in Randa(aaaaaaaa)

Planet KDE - Mon, 2017-09-18 20:00

Last week I had a chance to attend the Randa meetings 2017, my plan was to work on the Plasma Mobile during the sprint, improve the state of current images. A few things changed over the course of the week:

New shiny wallpaper and packages

Starting from 8 Sept, we started using the packages from KDE Neon buildsystem instead of building them on the Plasma Mobile CI, this provides few benefits,

  • No need to separately build the common things like Qt5, KDE Frameworks, Plasma
  • Always up-to-date packages, no need to manually re-trigger builds for common things

We also upgraded to newer version of Qt, Qt 5.9 from the older version of Qt 5.7.1, this aligns our stack with whatever KDE Neon dev unstable edition is using.

Rootfs for non-nexus Qualcomm devices

Non-nexus Qualcomm devices use a modified version of the hwcomposer.h which is not ABI compatible with the generic version of the android API headers, so for them the libhybris and kwin need to be separately built. Previously our CI infrastructure didn’t support this workflow, however now we are able to create the rootfs for such devices. Below is an image of Plasma Mobile running on Xiomi Mi device.

Marble maps

At Randa, we had a chance to test the newer version of the Marble maps, which has much more improved performance when zooming out-in. We were able to happily search locate Randa in the Marble maps.

QML based mobile friendly “konsole”

Previously we were using Konsole from the KDE Applications as a terminal emulator, however Konsole user interface was not mobile-friendly, and it was not possible to e.g, tab complete the command, go through history, etc. At Randa Emmanuel Lepage Vallee pointed us to qmltermwidget, which powers the terminal application in Ubuntu Touch and Cool-retro-term application. Marco Martin was able to write a simple QML application around qmltermwidget which can replace Konsole in Plasma Mobile.

Kirigami based koko

During GSoC 2017 lot of work has been put to port the normal QtQuick based image viewer application Koko to kirigami2, you can follow the work of Student developer Atul Sharma on his blog. At the Randa Sprint, we were able to test this work on Plasma Mobile and it shows great improvements compared to the older version.

Improved window switching experience

Due to a bug in the Plasma Mobile shell, shell windows such as task switcher and shell itself were also appearing in the task switcher, as a result of that user was able to close the shell window which ultimately ends up crashing the plasmashell. We were able to fix this bug in the plasma-workspace and plasma-phone-components so that Plasma Mobile shell windows would not appear in the task switcher.

Kube(!!!)

Yes, that is Kube in the first picture, we were able to run the Kube on the Plasma Mobile, it works in theory on Plasma Mobile, however the user interface is not mobile-friendly and needs various fixes to be usable on mobile. We will work with the Kube team to fix the issues and then test them on Plasma Mobile in the future.

Conclusion

That is all for now, I would like to thank the all the donors who donated for the Randa Meetings 2017, and made it possible for me to travel to Randa, Switzerland.

If you liked the work done on Plasma Mobile during the Randa Sprint, please support us on Randa 2017 fundraiser to make events like Randa possible in the future.

Categories: FLOSS Project Planets

Justin Mason: Links for 2017-09-18

Planet Apache - Mon, 2017-09-18 19:58
  • Native Memory Tracking

    Java 8 HotSpot feature to monitor and diagnose native memory leaks

    (tags: java jvm memory native-memory malloc debugging coding nmt java-8 jcmd)

  • This Heroic Captain Defied His Orders and Stopped America From Starting World War III

    Captain William Bassett, a USAF officer stationed at Okinawa on October 28, 1962, can now be added alongside Stanislav Petrov to the list of people who have saved the world from WWIII:

    By [John] Bordne’s account, at the height of the Cuban Missile Crisis, Air Force crews on Okinawa were ordered to launch 32 missiles, each carrying a large nuclear warhead. […] The Captain told Missile Operations Center over the phone that he either needed to hear that the threat level had been raised to DEFCON 1 and that he should fire the nukes, or that he should stand down. We don’t know exactly what the Missile Operations Center told Captain Bassett, but they finally received confirmation that they should not launch their nukes. After the crisis had passed Bassett reportedly told his men: “None of us will discuss anything that happened here tonight, and I mean anything. No discussions at the barracks, in a bar, or even here at the launch site. You do not even write home about this. Am I making myself perfectly clear on this subject?”

    (tags: wwiii history nukes cuban-missile-crisis 1960s usaf okinawa missiles william-bassett)

  • malware piggybacking on CCleaner

    On September 13, 2017 while conducting customer beta testing of our new exploit detection technology, Cisco Talos identified a specific executable which was triggering our advanced malware protection systems. Upon closer inspection, the executable in question was the installer for CCleaner v5.33, which was being delivered to endpoints by the legitimate CCleaner download servers. Talos began initial analysis to determine what was causing this technology to flag CCleaner. We identified that even though the downloaded installation executable was signed using a valid digital signature issued to Piriform, CCleaner was not the only application that came with the download. During the installation of CCleaner 5.33, the 32-bit CCleaner binary that was included also contained a malicious payload that featured a Domain Generation Algorithm (DGA) as well as hardcoded Command and Control (C2) functionality. We confirmed that this malicious version of CCleaner was being hosted directly on CCleaner’s download server as recently as September 11, 2017.

    (tags: ccleaner malware avast piriform windows security)

Categories: FLOSS Project Planets

Plasma 5.11 beta available in unofficial PPA for testing on Artful

Planet KDE - Mon, 2017-09-18 19:12

Adventurous users and developers running the Artful development release can now also test the beta version of Plasma 5.11. This is experimental and can possibly kill kittens!

Bug reports on this beta go to https://bugs.kde.org, not to Launchpad.

The PPA comes with a WARNING: Artful will ship with Plasma 5.10.5, so please be prepared to use ppa-purge to revert changes. Plasma 5.11 will ship too late for inclusion in Kubuntu 17.10, but should be available via the main backports PPA as soon as is practical after release day, October 19th, 2017.

Read more about the beta release: https://www.kde.org/announcements/plasma-5.10.95.php

If you want to test on Artful: sudo add-apt-repository ppa:kubuntu-ppa/beta && sudo apt-get update && sudo apt full-upgrade -y

The purpose of this PPA is testing, and bug reports go to bugs.kde.org.

Categories: FLOSS Project Planets

Carl Chenet: The Github threat

Planet Python - Mon, 2017-09-18 18:00

Many voices arise now and then against risks linked to the Github use by Free Software projects. Yet the infatuation for the collaborative forge of the Octocat Californian start-ups doesn’t seem to fade away.

These recent years, Github and its services take an important role in software engineering as they are seen as easy to use, efficient for a daily workload with interesting functions in enterprise collaborative workflow or amid a Free Software project. What are the arguments against using its services and are they valid? We will list them first, then we’ll examine their validity.

1. Critical points 1.1 Centralization

The Github application belongs to a single entity, Github Inc, a US company which manage it alone. So, a unique company under US legislation manages the access to most of Free Software application code sources, which may be a problem with groups using it when a code source is no longer available, for political or technical reason.

The Octocat, the Github mascot

 

This centralization leads to another trouble: as it obtained critical mass, it becomes more and more difficult not having a Github account. People who don’t use Github, by choice or not, are becoming a silent minority. It is now fashionable to use Github, and not doing so is seen as “out of date”. The same phenomenon is a classic, and even the norm, for proprietary social networks (Facebook, Twitter, Instagram).

1.2 A Proprietary Software

When you interact with Github, you are using a proprietary software, with no access to its source code and which may not work the way you think it is. It is a problem at different levels. First, ideologically, but foremost in practice. In the Github case, we send them code we can control outside of their interface. We also send them personal information (profile, Github interactions). And mostly, Github forces any project which goes through the US platform to use a crucial proprietary tools: its bug tracking system.

Windows, the epitome of proprietary software, even if others took the same path

 

1.3 The Uniformization

Working with Github interface seems easy and intuitive to most. Lots of companies now use it as a source repository, and many developers leaving a company find the same Github working environment in the next one. This pervasive presence of Github in free software development environment is a part of the uniformization of said developers’ working space.

Uniforms always bring Army in my mind, here the Clone army

2 – Critical points cross-examination 2.1 Regarding the centralization 2.1.1 Service availability rate

As said above, nowadays, Github is the main repository of Free Software source code. As such it is a favorite target for cyberattacks. DDOS hit it in March and August 2015. On December 15, 2015, an outage led to the inaccessibility of 5% of the repositories. The same occurred on November 15. And these are only the incident reported by Github itself. One can imagine that the mean outage rate of the platform is underestimated.

2.1.2 Chain reaction could block Free Software development

Today many dependency maintenance tools, as npm for javascript, Bundler for Ruby or even pip for Python can access an application source code directly from Github. Free Software projects getting more and more linked and codependents, if one component is down, all the developing process stop.

One of the best examples is the npmgate. Any company could legally demand that Github take down some source code from its repository, which could create a chain reaction and blocking the development of many Free Software projects, as suffered the Node.js community from the decisions of Npm, Inc, the company managing npm.

2.2 A historical precedent: SourceForge

Github didn’t appear out of the blue. In his time, its predecessor, SourceForge, was also extremely popular.

Heavily centralized, based on strong interaction with the community, SourceForge is now seen as an aging SAAS (Software As A Service) and sees most of its customers fleeing to Github. Which creates lots of hurdles for those who stayed. The Gimp project suffered from spams and terrible advertising, which led to the departure of the VLC project, then from installers corrupted with adwares instead of the official Gimp installer for Windows. And finally, the Project Gimp’s SourceForge account was hacked by… SourceForge team itself!

These are very recent examples of what can do a commercial entity when it is under its stakeholders’ pressure. It is vital to really understand what it means to trust them with data and exchange centralization, where it could have tremendous repercussion on the day-to-day life and the habits of the Free Software and open source community.

2.3. Regarding proprietary software 2.3.1 One community, several opinions on proprietary software

Mostly based on ideology, this point deals with the definition every member of the community gives to Free Software and open source. Mostly about one thing: is it viral or not? Or GPL vs MIT/BSD.

Those on the side of the viral Free Software will have trouble to use a proprietary software as this last one shouldn’t even exist. It must be assimilated, to quote Star Trek, as it is a connected black box, endangering privacy, corrupting for profit our uses and restrain our freedom to use as we’re pleased what we own, etc.

Those on the side of complete freedom have no qualms using proprietary software as their very existence is a consequence of freedom without restriction. They even agree that code they developed may be a part of proprietary software, which is quite a common occurrence. This part of the Free Software community has no qualm using Github, which is well within their ideology parameters. Just take a look at the Janson amphitheater during Fosdem and check how many Apple laptops running on macOS are around.

FreeBSD, the main BSD project under the BSD license

2.3.2 Data loss and data restrictions linked to proprietary software use

Even without ideological consideration, and just focusing on Github infrastructure, the bug tracking system is a major problem by itself.

Bug report builds the memory of Free Software projects. It is the entrance point for new contributors, the place to find bug reporting, requests for new functions, etc. The project history can’t be limited only to the code. It’s very common to find bug reports when you copy and paste an error message in a search engine. Not their historical importance is precious for the project itself, but also for its present and future users.

Github gives the ability to extract bug reports through its API. What would happen if Github is down or if the platform doesn’t support this feature anymore? In my opinion, not that many projects ever thought of this outcome. How could they move all the data generated by Github into a new bug tracking system?

One old example now is Astrid, a TODO list bought by Yahoo a few years ago. Very popular, it grew fast until it was closed overnight, with only a few weeks for its users to extract their data. It was only a to-do list. The same situation with Github would be tremendously difficult to manage for several projects if they even have the ability to deal with it. Code would still be available and could still live somewhere else, but the project memory would be lost. A project like Debian has today more than 800,000 bug reports, which are a data treasure trove about problems solved, function requests and where the development stand on each. The developers of the Cpython project have anticipated the problem and decided not to use Github bug tracking systems.

Issues, the Github proprietary bug tracking system

Another thing we could lose if Github suddenly disappear: all the work currently done regarding the push requests (aka PRs). This Github function gives the ability to clone one project’s Github repository, to modify it to fit your needs, then to offer your own modification to the original repository. The original repository’s owner will then review said modification, and if he or she agrees with them will fuse them into the original repository. As such, it’s one of the main advantages of Github, since it can be done easily through its graphic interface.

However reviewing all the PRs may be quite long, and most of the successful projects have several ongoing PRs. And this PRs and/or the proprietary bug tracking system are commonly used as a platform for comment and discussion between developers.

Code itself is not lost if Github is down (except one specific situation as seen below), but the peer review works materialized in the PRs and the bug tracking system is lost. Let’s remember than the PR mechanism let you clone and modify projects and then generate PRs directly from its proprietary web interface without downloading a single code line on your computer. In this particular case, if Github is down, all the code and the work in progress is lost.

Some also use Github as a bookmark place. They follow their favorite projects’ activity through the Watch function. This technological watch style of data collection would also be lost if Github is down.

Debian, one of the main Free Software projects with at least a thousand official contributors

2.4 Uniformization

The Free Software community is walking a thigh rope between normalization needed for an easier interoperability between its products and an attraction for novelty led by a strong need for differentiation from what is already there.

Github popularized the use of Git, a great tool now used through various sectors far away from its original programming field. Step by step, Git is now so prominent it’s almost impossible to even think to another source control manager, even if awesome alternate solutions, unfortunately not as popular, exist as Mercurial.

A new Free Software project is now a Git repository on Github with README.md added as a quick description. All the other solutions are ostracized? How? None or very few potential contributors would notice said projects. It seems very difficult now to encourage potential contributors into learning a new source control manager AND a new forge for every project they want to contribute. Which was a basic requirement a few years ago.

It’s quite sad because Github, offering an original experience to its users, cut them out of a whole possibility realm. Maybe Github is one of the best web versioning control systems. But being the main one doesn’t let room for a new competitor to grow. And it let Github initiate development newcomers into a narrow function set, totally unrelated to the strength of the Git tool itself.

3. Centralization, uniformization, proprietary software… What’s next? Laziness?

Fight against centralization is a main part of the Free Software ideology as centralization strengthens the power of those who manage it and who through it control those who are managed by it. Uniformization allergies born against main software companies and their wishes to impose a closed commercial software world was for a long time the main fuel for innovation thirst and intelligent alternative development. As we said above, part of the Free Software community was built as a reaction to proprietary software and their threat. The other part, without hoping for their disappearance, still chose a development model opposite to proprietary software, at least in the beginning, as now there’s more and more bridges between the two.

The Github effect is a morbid one because of its consequences: at least centralization, uniformization, proprietary software usage as their bug tracking system. But some years ago the Dear Github buzz showed one more side effect, one I’ve never thought about: laziness. For those who don’t know what it is about, this letter is a complaint from several spokespersons from several Free Software projects which demand to Github team to finally implement, after years of polite asking, new functions.

Since when Free Software project facing a roadblock request for clemency and don’t build themselves the path they need? When Torvalds was involved in the Bitkeeper problem and the Linux kernel development team couldn’t use anymore their revision control software, he developed Git. The mere fact of not being able to use one tool or functions lacking is the main motivation to seek alternative solutions and, as such, of the Free Software movement. Every Free Software community member able to code should have this reflex. You don’t like what Github offers? Switch to Gitlab. You don’t like it Gitlab? Improve it or make your own solution.

The Gitlab logo

Let’s be crystal clear. I’ve never said that every Free Software developers blocked should code his or her own alternative. We all have our own priorities, and some of us even like their beauty sleep, including me. But, to see that this open letter to Github has 1340 names attached to it, among them some spokespersons for major Free Software project showed me that need, willpower and strength to code a replacement are here. Maybe said replacement will be born from this letter, it would be the best outcome of this buzz.

In the end, Github usage is just another example of Internet usage massification. As Internet users are bound to go to massively centralized social network as Facebook or Twitter, developers are following the same path with Github. Even if a large fraction of developers realize the threat linked this centralized and proprietary organization, the whole community is following this centralization and uniformization trend. Github service is useful, free or with a reasonable price (depending on the functions you need) easy to use and up most of the time. Why would we try something else? Maybe because others are using us while we are savoring the convenience? The Free Software community seems to be quite sleepy to me.

The lion enjoying the hearth warm

About Me

Carl Chenet, Free Software Indie Hacker, founder of the French-speaking Hacker News-like Journal du hacker.

Follow me on social networks

Translated from French by Stéphanie Chaptal. Original article written in 2015.

Categories: FLOSS Project Planets

Carl Chenet: The Github threat

Planet Debian - Mon, 2017-09-18 18:00

Many voices arise now and then against risks linked to the Github use by Free Software projects. Yet the infatuation for the collaborative forge of the Octocat Californian start-ups doesn’t seem to fade away.

These recent years, Github and its services take an important role in software engineering as they are seen as easy to use, efficient for a daily workload with interesting functions in enterprise collaborative workflow or amid a Free Software project. What are the arguments against using its services and are they valid? We will list them first, then we’ll examine their validity.

1. Critical points 1.1 Centralization

The Github application belongs to a single entity, Github Inc, a US company which manage it alone. So, a unique company under US legislation manages the access to most of Free Software application code sources, which may be a problem with groups using it when a code source is no longer available, for political or technical reason.

The Octocat, the Github mascot

 

This centralization leads to another trouble: as it obtained critical mass, it becomes more and more difficult not having a Github account. People who don’t use Github, by choice or not, are becoming a silent minority. It is now fashionable to use Github, and not doing so is seen as “out of date”. The same phenomenon is a classic, and even the norm, for proprietary social networks (Facebook, Twitter, Instagram).

1.2 A Proprietary Software

When you interact with Github, you are using a proprietary software, with no access to its source code and which may not work the way you think it is. It is a problem at different levels. First, ideologically, but foremost in practice. In the Github case, we send them code we can control outside of their interface. We also send them personal information (profile, Github interactions). And mostly, Github forces any project which goes through the US platform to use a crucial proprietary tools: its bug tracking system.

Windows, the epitome of proprietary software, even if others took the same path

 

1.3 The Uniformization

Working with Github interface seems easy and intuitive to most. Lots of companies now use it as a source repository, and many developers leaving a company find the same Github working environment in the next one. This pervasive presence of Github in free software development environment is a part of the uniformization of said developers’ working space.

Uniforms always bring Army in my mind, here the Clone army

2 – Critical points cross-examination 2.1 Regarding the centralization 2.1.1 Service availability rate

As said above, nowadays, Github is the main repository of Free Software source code. As such it is a favorite target for cyberattacks. DDOS hit it in March and August 2015. On December 15, 2015, an outage led to the inaccessibility of 5% of the repositories. The same occurred on November 15. And these are only the incident reported by Github itself. One can imagine that the mean outage rate of the platform is underestimated.

2.1.2 Chain reaction could block Free Software development

Today many dependency maintenance tools, as npm for javascript, Bundler for Ruby or even pip for Python can access an application source code directly from Github. Free Software projects getting more and more linked and codependents, if one component is down, all the developing process stop.

One of the best examples is the npmgate. Any company could legally demand that Github take down some source code from its repository, which could create a chain reaction and blocking the development of many Free Software projects, as suffered the Node.js community from the decisions of Npm, Inc, the company managing npm.

2.2 A historical precedent: SourceForge

Github didn’t appear out of the blue. In his time, its predecessor, SourceForge, was also extremely popular.

Heavily centralized, based on strong interaction with the community, SourceForge is now seen as an aging SAAS (Software As A Service) and sees most of its customers fleeing to Github. Which creates lots of hurdles for those who stayed. The Gimp project suffered from spams and terrible advertising, which led to the departure of the VLC project, then from installers corrupted with adwares instead of the official Gimp installer for Windows. And finally, the Project Gimp’s SourceForge account was hacked by… SourceForge team itself!

These are very recent examples of what can do a commercial entity when it is under its stakeholders’ pressure. It is vital to really understand what it means to trust them with data and exchange centralization, where it could have tremendous repercussion on the day-to-day life and the habits of the Free Software and open source community.

2.3. Regarding proprietary software 2.3.1 One community, several opinions on proprietary software

Mostly based on ideology, this point deals with the definition every member of the community gives to Free Software and open source. Mostly about one thing: is it viral or not? Or GPL vs MIT/BSD.

Those on the side of the viral Free Software will have trouble to use a proprietary software as this last one shouldn’t even exist. It must be assimilated, to quote Star Trek, as it is a connected black box, endangering privacy, corrupting for profit our uses and restrain our freedom to use as we’re pleased what we own, etc.

Those on the side of complete freedom have no qualms using proprietary software as their very existence is a consequence of freedom without restriction. They even agree that code they developed may be a part of proprietary software, which is quite a common occurrence. This part of the Free Software community has no qualm using Github, which is well within their ideology parameters. Just take a look at the Janson amphitheater during Fosdem and check how many Apple laptops running on macOS are around.

FreeBSD, the main BSD project under the BSD license

2.3.2 Data loss and data restrictions linked to proprietary software use

Even without ideological consideration, and just focusing on Github infrastructure, the bug tracking system is a major problem by itself.

Bug report builds the memory of Free Software projects. It is the entrance point for new contributors, the place to find bug reporting, requests for new functions, etc. The project history can’t be limited only to the code. It’s very common to find bug reports when you copy and paste an error message in a search engine. Not their historical importance is precious for the project itself, but also for its present and future users.

Github gives the ability to extract bug reports through its API. What would happen if Github is down or if the platform doesn’t support this feature anymore? In my opinion, not that many projects ever thought of this outcome. How could they move all the data generated by Github into a new bug tracking system?

One old example now is Astrid, a TODO list bought by Yahoo a few years ago. Very popular, it grew fast until it was closed overnight, with only a few weeks for its users to extract their data. It was only a to-do list. The same situation with Github would be tremendously difficult to manage for several projects if they even have the ability to deal with it. Code would still be available and could still live somewhere else, but the project memory would be lost. A project like Debian has today more than 800,000 bug reports, which are a data treasure trove about problems solved, function requests and where the development stand on each. The developers of the Cpython project have anticipated the problem and decided not to use Github bug tracking systems.

Issues, the Github proprietary bug tracking system

Another thing we could lose if Github suddenly disappear: all the work currently done regarding the push requests (aka PRs). This Github function gives the ability to clone one project’s Github repository, to modify it to fit your needs, then to offer your own modification to the original repository. The original repository’s owner will then review said modification, and if he or she agrees with them will fuse them into the original repository. As such, it’s one of the main advantages of Github, since it can be done easily through its graphic interface.

However reviewing all the PRs may be quite long, and most of the successful projects have several ongoing PRs. And this PRs and/or the proprietary bug tracking system are commonly used as a platform for comment and discussion between developers.

Code itself is not lost if Github is down (except one specific situation as seen below), but the peer review works materialized in the PRs and the bug tracking system is lost. Let’s remember than the PR mechanism let you clone and modify projects and then generate PRs directly from its proprietary web interface without downloading a single code line on your computer. In this particular case, if Github is down, all the code and the work in progress is lost.

Some also use Github as a bookmark place. They follow their favorite projects’ activity through the Watch function. This technological watch style of data collection would also be lost if Github is down.

Debian, one of the main Free Software projects with at least a thousand official contributors

2.4 Uniformization

The Free Software community is walking a thigh rope between normalization needed for an easier interoperability between its products and an attraction for novelty led by a strong need for differentiation from what is already there.

Github popularized the use of Git, a great tool now used through various sectors far away from its original programming field. Step by step, Git is now so prominent it’s almost impossible to even think to another source control manager, even if awesome alternate solutions, unfortunately not as popular, exist as Mercurial.

A new Free Software project is now a Git repository on Github with README.md added as a quick description. All the other solutions are ostracized? How? None or very few potential contributors would notice said projects. It seems very difficult now to encourage potential contributors into learning a new source control manager AND a new forge for every project they want to contribute. Which was a basic requirement a few years ago.

It’s quite sad because Github, offering an original experience to its users, cut them out of a whole possibility realm. Maybe Github is one of the best web versioning control systems. But being the main one doesn’t let room for a new competitor to grow. And it let Github initiate development newcomers into a narrow function set, totally unrelated to the strength of the Git tool itself.

3. Centralization, uniformization, proprietary software… What’s next? Laziness?

Fight against centralization is a main part of the Free Software ideology as centralization strengthens the power of those who manage it and who through it control those who are managed by it. Uniformization allergies born against main software companies and their wishes to impose a closed commercial software world was for a long time the main fuel for innovation thirst and intelligent alternative development. As we said above, part of the Free Software community was built as a reaction to proprietary software and their threat. The other part, without hoping for their disappearance, still chose a development model opposite to proprietary software, at least in the beginning, as now there’s more and more bridges between the two.

The Github effect is a morbid one because of its consequences: at least centralization, uniformization, proprietary software usage as their bug tracking system. But some years ago the Dear Github buzz showed one more side effect, one I’ve never thought about: laziness. For those who don’t know what it is about, this letter is a complaint from several spokespersons from several Free Software projects which demand to Github team to finally implement, after years of polite asking, new functions.

Since when Free Software project facing a roadblock request for clemency and don’t build themselves the path they need? When Torvalds was involved in the Bitkeeper problem and the Linux kernel development team couldn’t use anymore their revision control software, he developed Git. The mere fact of not being able to use one tool or functions lacking is the main motivation to seek alternative solutions and, as such, of the Free Software movement. Every Free Software community member able to code should have this reflex. You don’t like what Github offers? Switch to Gitlab. You don’t like it Gitlab? Improve it or make your own solution.

The Gitlab logo

Let’s be crystal clear. I’ve never said that every Free Software developers blocked should code his or her own alternative. We all have our own priorities, and some of us even like their beauty sleep, including me. But, to see that this open letter to Github has 1340 names attached to it, among them some spokespersons for major Free Software project showed me that need, willpower and strength to code a replacement are here. Maybe said replacement will be born from this letter, it would be the best outcome of this buzz.

In the end, Github usage is just another example of Internet usage massification. As Internet users are bound to go to massively centralized social network as Facebook or Twitter, developers are following the same path with Github. Even if a large fraction of developers realize the threat linked this centralized and proprietary organization, the whole community is following this centralization and uniformization trend. Github service is useful, free or with a reasonable price (depending on the functions you need) easy to use and up most of the time. Why would we try something else? Maybe because others are using us while we are savoring the convenience? The Free Software community seems to be quite sleepy to me.

The lion enjoying the hearth warm

About Me

Carl Chenet, Free Software Indie Hacker, founder of the French-speaking Hacker News-like Journal du hacker.

Follow me on social networks

Translated from French by Stéphanie Chaptal. Original article written in 2015.

Categories: FLOSS Project Planets

Christopher Allan Webber: DRM will unravel the Web

GNU Planet! - Mon, 2017-09-18 15:30

I'm a web standards author and I participate in the W3C. I am co-editor of the ActivityPub protocol, participate in a few other community groups and working groups, and I consider it an honor to have been able to participate in the W3C process. What I am going to write here though represents me and my feelings alone. In a sense though, that makes this even more painful. This is a blogpost I don't have time to write, but here I am writing it; I am emotionally forced to push forward on this topic. The W3C has allowed DRM to move forward on the web through the EME specification (which is, to paraphrase Danny O'Brien from the EFF, a "DRM shaped hole where nothing else but DRM fits"). This threatens to unravel the web as we know it. How could this happen? How did we get here?

Like many of my generation, I grew up on the web, both as a citizen of this world and as a developer. "Web development", in one way or another, has principally been my work for my adult life, and how I have learned to be a programmer. The web is an enormous, astounding effort of many, many participants. Of course, Tim Berners-Lee is credited for much of it, and deserves much of this credit. I've had the pleasure of meeting Tim on a couple of occasions; when you meet Tim it's clear how deeply he cares about the web. Tim speaks quickly, as though he can't wait to get out the ideas that are so important to him, to try to help you understand how wonderful and exciting this system it is that we can build together. Then, as soon as he's done talking, he returns to his computer and gets to hacking on whatever software he's building to advance the web. You don't see this dedication to "keep your hands dirty" in the gears of the system very often, and it's a trait I admire. So it's very hard to reconcile that vision of Tim with someone who would intentionally unravel their own work... yet by allowing the W3C to approve DRM/EME, I believe that's what has happened.

I had an opportunity to tell Tim what I think about DRM and EME on the web, and unfortunately I blew it. At TPAC (W3C's big conference/gathering of the standards minds) last year, there was a protest against DRM outside. I was too busy to take part, but I did talk to a friend who is close to Tim and was frustrated about the protests happening outside. After I expressed that I sympathized with the protestors (and that I had even indeed protested myself in Boston), I explained my position to my friend. Apparently I was convincing enough where they encouraged me to talk to Tim and offer my perspective; they offered to flag them down for a chat. In fact Tim and I did speak over lunch, but -- although we had met in person before -- it was my first time talking to Tim one-on-one, and I was embarassed for that first interaction would me to be talking about DRM and what I was afraid was a sore subject for him. Instead we had a very pleasant conversation about the work I was doing on ActivityPub and some related friends' work on other standards (such as Linked Data Notifications, etc). It was a good conversation, but when it was over I had an enormous feeling of regret that has been on the back of my mind since.

Here then, is what I wish I had said.

Tim, I have read your article on why the W3C is supporting EME, and that I know you have thought about it a great deal. I think you believe what you are doing what is right for the web, but I believe you are making an enormous miscalculation. You have fought long and hard to build the web into the system it is... unfortunately, I think DRM threatens to undo all that work so thoroughly that allowing the W3C to effectively green-light DRM for the web will be, looking back on your life, your greatest regret.

You and I both know the dangers of DRM: it creates content that is illegal to operate on using any of the tooling you or I will ever be able to write. The power of DRM is not in its technology but in the surrounding laws; in the United States through the DMCA it is a criminal offense to inspect how DRM systems work or to talk about these vulnerabilities. DRM is also something that clearly cannot itself be implemented as a standard; it relies on proprietary secrecy in order to be able to function. Instead, EME defines a DRM-shaped hole, but we all know what goes into that hole... unfortunately, there's no way for you or I to build an open and interoperable system that can fit in that EME hole, because DRM is antithetical to an interoperable, open web.

I think, from reading your article, that you believe that DRM will be safely contained to just "premium movies", and so on. Perhaps if this were true, DRM would still be serious but not as enormous of a threat as I believe it is. In fact, we already know that DRM is being used by companies like John Deere to say that you don't even own your own tractor, car, etc. If DRM can apply to tractors, surely it will apply to more than just movies.

Indeed, there's good reason to believe that some companies will want to apply DRM to every layer of the web. Since the web has become a full-on "application delivery system", of course the same companies that apply DRM to software will want to apply DRM to their web software. The web has traditionally been a book which encourages being opened; I learned much of how to program on the web through that venerable "view source" right-click menu item of web browsers. However I fully expect with EME that we will see application authors begin to lock down HTML, CSS, Javascript, and every other bit of their web applications down with DRM. (I suppose in a sense this is already happening with javascript obfuscation and etc, but the web itself was at least a system of open standards where anyone could build an implementation and anyone could copy around files... with EME, this is no longer the case.) Look at the prevelance of DRM in proprietary applications elsewhere... once the option of a W3C-endorsed DRM-route exists, do you think these same application developers will not reach for it? But I think if you develop the web with the vision of it being humanity's greatest and most empowering knowledge system, you must be against this, because if enough of the web moves over to this model the assumptions and properties of the web as we've known it, as an open graph to free the world, cannot be upheld. I also know the true direction you'd like the web to go, one of linked data systems (of which ActivityPub is somewhat quietly one). Do you think such a world will be possible to build with DRM? I for one do not see how it is possible, but I'm afraid that's the path down which we are headed.

I'm sure you've thought of these things too, so what could be your reason for deciding to go ahead with supporting DRM anyway? My suspicion is it's two things contributing to this:

  1. Fear that the big players will pick up their ball and leave. I suspect there's fear of another WHATWG, that the big players will simply pick up their ball and leave.
  2. Most especially, and related to the above, I suspect the funding and membership structure of the W3C is having a large impact on this. Funding structures tend to have a large impact on decision making, as a kind of Conway's Law effect. W3C is reliant on its "thin gruel" of funding from member organizations (which means that large players tend to have a larger say in how the web is built today).

I suspect this is most of all what's driving the support for DRM within the W3C. However, I know a few W3C staff members who are clearly not excited about DRM, and two who have quit the organization over it, so it's not that EME is internally a technology that brings excitement to the organziation.

I suppose at this point, this is where I diverge with the things I could have said in the past and did not say as an appeal to not allow the W3C to endorse EME. Unfortunately, today EME made it to Recommendation. At the very least, I think the W3C could have gone forward with the Contributor Covenant proposed by the EFF, but did not. This is an enormous disappointment.

What do we do now? I think the best we can do at this point, as individual developers and users, is speak out against DRM and refuse to participate in it.

And Tim, if you're listening, perhaps there's no chance now to stop EME from becoming a Recommendation. But your voice can still carry weight. I encourage you to join in speaking out against the threat DRM brings to unravel the web.

Perhaps if we speak loud enough, and push hard enough, we can still save the web we love. But today is a sad say, and from here I'm afraid it is going to be an uphill battle.

EDIT: If you haven't yet read Cory Doctorow / the EFF's open letter to the W3C, you should.

Categories: FLOSS Project Planets
Syndicate content