FLOSS Project Planets

Import Python: ImportPython Issue 82

Planet Python - Tue, 2016-10-25 03:39

Worthy Read
Python has come a long way. So has job hunting.
Try Hired and get in front of 4,000+ companies with one application. No more pushy recruiters, no more dead end applications and mismatched companies, Hired puts the power in your hands.Sponsor
Machine Learning over 1M hotel reviews finds interesting insightsOn this tutorial we learned how to scrape millions of reviews, analyze them with pre-trained classifiers within MonkeyLearn, indexed the results with Elasticsearch and visualize them using Kibana. Machine learning makes sense when you want to analyze big volumes of data in a cost effective way. The code repository is here - https://github.com/monkeylearn/hotel-review-analysis

Mike Driscoll: Python 201: An Intro to mockThe unittest module now includes a mock submodule as of Python 3.3. It will allow you to replace portions of the system that you are testing with mock objects as well as make assertions about how they were used. A mock object is used for simulating system resources that aren’t available in your test environment. In other words, you will find times when you want to test some part of your code in isolation from the rest of it or you will need to test some code in isolation from outside services.

Altair: Declarative statistical visualization library for Python, based on Vega-Litepep8Altair is a declarative statistical visualization library for Python.

7 Django Development Best Practices Each Web Developer Must KnowdjangoSet up Persistent Database Connections, Turn Cached Loading on, Store the Sessions in Cache, Keep the Application and Libraries Separate, Store All Templates in One Place, Install HTML5 Boilerplate, Monitor and Control Processes using Supervisor.

DSF Code of Conduct committee releases transparent documentationcommunityToday we're proud to open source the documentation that describes how the Django Code of Conduct committee enforces our Code of Conduct. This documentation covers the structure of Code of Conduct committee membership, the process of handling Code of Conduct violations, our decision making process, record keeping, and transparency.

Why are some functions in python spelled with underscore, while some are not: setdefault, makedirs, isinstance?discussionI always wondered that. Here is a reddit discussion on the same.

Teaching an AI to write Python code with Python codeAIThis post is about creating a machine that writes its own code. More or less. Introducing GlaDoS Skynet Spynet. More specifically, we are going to train a character level Long Short Term Memory neural network to write code itself by feeding it Python source code. The training will run on a GPU instance on EC2, using Theano and Lasagne. If some of the words here sound obscure to you, I will do my best to explain what is happening.

Writing an API with Flask-RESTfulRESTThis article will go over the details of how to create a RESTful API with Flask and Flask-RESTful. In Part 1 we will go over the API basics and how to implement a simple API. In Part 2 we will expand into advanced use cases powered by Flask-RESTful. All code that will be show is readily available on this repository.

SciPy 2016 videos are upvideoRunning Python Apps in the Browser by Almar Klein was a pretty interesting talk for me. See what interest you. Youtube channel.

How to Create a Custom Django MiddlewaredjangoIn a nutshell, a Middleware is a regular Python class that hooks into Django’s request/response life cycle. Those classes holds pieces of code that are processed upon every request/response your Django application handles.

Mike Driscoll: An Intro to coverage.pycoverageCoverage.py is a 3rd party tool for Python that is used for measuring your code coverage. It was originally created by Ned Batchelder. The term “coverage” in programming circles is typically used to describe the effectiveness of your tests and how much of your code is actually covered by tests. You can use coverage.py with Python 2.6 up to the current version of Python 3 as well as with PyPy.

Ajax Website Tutorial with DjangodjangoIn this tutorial we'll see a trivial example of how to do a ajax website with django. Good for students looking to learn the basics of Django/Ajax and see how it works.

Check out the Python & Django channels available on Gitter.communityGitter is like slack for developers. They have active Python, Django channels. Have a look.

Introduction to Zipline in PythonPython has emerged as one of the most popular language for programmers in financial trading, due to its ease of availability, user-friendliness and presence of sufficient scientific libraries like Pandas, NumPy, PyAlgoTrade, Pybacktest and more. Zipline is a Python library for trading applications that powers the Quantopian service mentioned above. It is an event-driven system that supports both backtesting and live-trading. In this article we will learn how to install Zipline and then how to implement Moving Average Crossover strategy and calculate P&L, Portfolio value etc.

Upcoming Conference / User Group Meet
PyCon Australia 2016
PyCon APAC 2016
EuroScipy 2016
PyCon MY 2016
Python Unconference 2016
Kiwi PyCon
PyCon ZA 2016

PokemonGo-DesktopMap - 204 Stars, 36 ForkElectron App around PokemonGo-Map
PokemonGo-Map - 128 Stars, 55 ForkLive visualization of all the pokemon in your area
asyncpg - 69 Stars, 2 ForkA fast PostgreSQL Database Client Library for Python/asyncio
choronzon - 46 Stars, 16 ForkAn evolutionary knowledge-based fuzzer
zhihu-terminal - 42 Stars, 2 Forkzhihu-terminal using python2.7.
awesome-wagtail - 14 Stars, 1 ForkA curated list of awesome packages, articles, and other cool resources from the Wagtail community.
reddit_get_top_images - 10 Stars, 1 ForkGet top images from any subreddit
aiosmtpd - 6 Stars, 1 ForkA reimplementation of the Python stdlib smtpd.py based on asyncio.
delft - 6 Stars, 1 ForkA Python tool that automatically optimizes deep learning pipelines using genetic programming.
Categories: FLOSS Project Planets

Import Python: ImportPython Issue 81

Planet Python - Tue, 2016-10-25 03:39

Worthy Read
Preview the Python Serverless Microframework for AWSawsServerless computing is one of the most talked-about subjects among AWS customers. The AWS serverless offerings, AWS Lambda and Amazon API Gateway, make it possible for developers to create and run API applications with built-in, virtually unlimited scalability without managing any servers. Today the AWS Developer Tools team is excited to announce the preview of the Python Serverless Microframework for AWS. You can read Martin Fowler talking about the benefits of Serverless architecture http://martinfowler.com/articles/serverless.html#benefits

Continuous Delivery Tool
Automate from SCM to Production with BuildMaster! 100+ Integrations; Free for 5 Users.Sponsor
Reverse debugging for PythonpypyThe PyPy team is pleased to give you "RevPDB", a reverse-debugger similar to rr but for Python.

Facebook chat bot aka joke bot with django tutorialchatbotsI have decided to try to develop a chat bot which does only one thing. Send a random joke like the below one without an image irrespective of what the user types

Book review: “Two Scoops of Django: Best Practices for Django 1.8”book reviewThe book can be used as a reference of best practices and a cover-to-cover guide to best practices. I’ve done both and found it to be enjoyable, accessible, and educational when read cover-to-cover and a valuable reference when setting up a new Django project or doing general Django development. It covers a huge range of material.

Using Spectrum - Standalone Logging Server with Python. ReviewSpectrum is a standalone logging server plus log viewer with filtering capabilities. It scales to multiple logging streams with endpoint being a file residing on filesystem, REST API endpoint, Syslog, UDPStream, WebSocketStream.

Create python app and deploy in cloud for free.
Have an app idea and want to see if it’ll fly ? Create a new Python app, free with Azure App Service. Build web, mobile, and API apps that use Redis Cache, Search, or Content Delivery Network and more. Check it out.Sponsor
Build Your First Python and Django ApplicationtutorialDecent tutorial to get people started with Python and Django.

Building an Async Python 3 Serverless Application with DockerserverlessI will show you a simple way to build a “serverless” application and test it via Docker. When I refer to “serverless” I’m referring to the idea that the application is a short lived app, does its job, stops – just like AWS Lambda. I will create two applications each in their own project folders: serverless-app and serverless-web The serverless-app piece is the actual “serverless” piece of this, the web app will run as long as we want. I just gave them similar names to make it easier to keep the projects named closely but different enough to know what does what.

djangobot: Bridge between Slack and Django, via ChannelsbotDjangobot is a bridge between Slack and a Channels-enabled Django app. Specifically, it is a protocol server that produces and consumes messages for channels-based apps. It is built atop autobahn and twisted.

Visualizing relationships between python packagespypiI extracted co-occurence of top 3500 python packages in github repos using the the github data on BigQuery. I implemented the visualization force layout in d3 via the velocity verlet integration. I also clustered the graph using algorithms from python-igraph and updated it to http://graphistry.com/.

Hiding a payload in PNG files with PythonsecurityHow could I store files online, in plain sight, for free. Because who doesn’t like a good ‘ol game of hide and seek. But with files. On the internet. Hide files in plain sight. Allow them to be distributed via free public channels. E.g Twitter, Reddit, imgur.


Senior Python Developer at EPAMMexicoEPAM Systems, Inc. (NYSE:EPAM), a leading global product development and platform engineering services company, is focused on delivering results through best in-class software engineering, combined with innovative strategy, consulting and design capabilities. Responsible for many phases of the software development lifecycle including technical research, and requirements analysis for Python projects.

Upcoming Conference / User Group Meet
PyGotham 2016
EuroPython 2016
Python Northwest
PyOhio 2016
PyCon Australia 2016
PyCon APAC 2016
EuroScipy 2016
PyCon MY 2016

aq - 31 Stars, 0 ForkQuery AWS resources with SQL. aq allows you to query your AWS resources (EC2 instances, S3 buckets, etc.) with plain SQL.
sokChoGo - 24 Stars, 5 ForkTravel the pokemon world in your comfy bed, with your keyboards, not on your legs.
macOSLAPS - 19 Stars, 2 ForkA python script that will change a local adminsitrator password to a random generated password. Similar behavior to LAPS for Windows
Pokemon_Go_API - 13 Stars, 0 ForkPokémon GO API in Python
stdgif - 10 Stars, 0 ForkStandard output for gifs. Dumps gifs to stdout or creates shell scripts that can be sourced from a .bashrc or other shell initialization file.
pstack - 9 Stars, 1 ForkTool to dump python thread and greenlet stacks. pstack is to python as jstack is to java! It's a debug tool to print python threads or greenlet stacks.
csv-sql - 8 Stars, 0 ForkQuery your CSV files with SQL
PostMail - 6 Stars, 1 ForkA simple mail server which can let you send a email only sending a POST request
Categories: FLOSS Project Planets

Import Python: Using Spectrum - Standalone Logging Server with Python. Review

Planet Python - Tue, 2016-10-25 03:39
Debugging, Logging and Software Development “ If debugging is the process of removing software bugs, then programming must be the process of putting them in. ” Edsger Dijkstra

During development of new features and integration there of a developer can be found staring at log data as it pours on his screen for errors, observing variable values and if they are in sync with expectation. Developers spends more time confronting, debugging and fixing software issues during development of software rather then after it has been deployed ( hopefully ).

Spectrum is to logging what sqlite3 is to databases. It’s a standalone logging server plus log viewer with filtering capabilities. It scales to multiple logging streams with endpoint being a file residing on filesystem, REST API endpoint, Syslog, UDPStream, WebSocketStream.

Let us give Spectrum a spin and see how it works.

Create Account

Visit https://www.devspectrum.com/register/ and register your account. This includes buying spectrum right away and sadly there is no trial period. Disclaimer we were provided access to the software by Spectrum for this product review.

Download Build and Registration Keys

Once your account is created the dashboard https://www.devspectrum.com/dashboard/ lists everything you need to get started

a) Copy Registration code somewhere secure,

b) Click “Download Latest Release” and select the operating system of your choice i.e. Linux, Windows, OSX.

c) Once the build has been downloaded extract the archive. (We are using the Linux build for this article). Execute Spectrum binary and you are greeted with a popup for registration code. Enter your Registration code from step 1 and we are in business.

Spectrum allows developer to organize logging into projects. Each project can have multiple streams. Each logging stream in turn can accommodates multiple levels of logging. Well that’s quite a mouthful but below examples will clear it.

File Based Stream

To fans of doing tail -f logfile.log on terminal this is for you. You wouldn’t have to worry about scrollback buffer. Simply create an endpoint with File as an option and select the log file.

Syslog and File based Stream import logging import logging.handlers logger = logging.getLogger(__name__) logger.setLevel(logging.DEBUG) handler = logging.handlers.SysLogHandler(address = '/dev/log') formatter = logging.Formatter('%(module)s.%(funcName)s: %(message)s') handler.setFormatter(formatter) logger.addHandler(handler) def dummylogs(): logger.debug('this is debug') logger.critical('this is critical') if __name__ == '__main__': dummylogs()

The entry level barrier to get spectrum up and running is very low, that’s why I called it sqlite3 of databases. Having the ability to search is very helpful too. e.g. “Traceback” shows the Python errors while the program executed.

REST based Stream

Spectrum can run as a standalone rest server binding to localhost and a port number of your choice. Python program can stream log messages to the server using spectrum-python library installable using pip.

pip install spectrum-python

Before we dive into code, it’s worth noting that a simple JSON HTTP POST to http://IP:PORT is all that’s needed to log messages to Spectrum in REST mode.

import requests url = "http://localhost:9000" data = {"id": "863ef2c3ea134841b87072", "level": "ERROR", "message": "The daemon process has generated an error code 555", "number": 253, "sublevel": "DAEMON", "timestamp": "2016-02-22 11:12:39.873108" } headers = {'Content-type': 'application/json', 'Accept': 'text/plain'} r = requests.post(url, data=json.dumps(data), headers=headers)

as long as your json has the keys id, level, sublevel, message, number, sublevel and timestamp you will be able to successfully log your message.

import logging from spectrum.handlers import Spectrum as SpectrumAPI logger = logging.getLogger(__name__) logger.setLevel(logging.DEBUG) spectrum = SpectrumAPI('Main-Process') formatter = logging.Formatter('%(module)s.%(funcName)s: %(message)s') spectrum.setFormatter(formatter) logger.addHandler(spectrum) def dummylogs(): logger.debug('this is debug') logger.critical('this is critical') if __name__ == '__main__': dummylogs() Conclusion Pros
  • Minimal Viable logger with a decent UI
  • Easy to install, integrate and use
  • Works out of the box for Python
  • Ability to accommodate N no of projects with N no of streams and loggers.
  • Search bar could have access to filters to toggle and not overlap the first line,
  • File based logging method froze and crashed on us couple of times. Had to remove .spectrum directory in home folder and reinstall. This could be attributed to the fact that spectrum is a chrome app and not good at keeping up with filesystem IO. The software was writing an MB worth of data to the log file per second.
Categories: FLOSS Project Planets

Import Python: ImportPython Issue 79

Planet Python - Tue, 2016-10-25 03:39

Worthy Read
Have an idea for a Python App ? Get FREE $200 in Azure credit to get started.
Create a new Python app, free with Azure App Service. Free $200 in Azure credit. Build web, mobile, and API apps that use Redis Cache, Search, or Content Delivery Network and more.Sponsor
Top 10 Django Projects Started in 2016djangoOne of the cornerstones of Django is that there’s a solid core that others can extend and share their extensions. So it’s good to celebrate the best of those. Here’s my list derived from this query on Github for projects started this year. Not sure about top but useful projects that one can use in personal or work projects/products. Have a look.

Python programming humourhumourConversation with Tech Support.

Supercharging Django ProductivitydjangoThese design patterns also solidified a mentality within our team that there is an internal standard of handling models and views. We can extend our base classes to tackle any business requirement. Our codebase is constantly evolving, both in new features and how we model the world. Django has proven to be a valuable tool that lets us aim big and iterate often. Some might call this an anti-pattern nevertheless a novel approach.

Distribute Tasks with Python Celery and RabbitMQceleryIn this tutorial, we are going to have an introduction to basic concepts of Celery with RabbitMQ and then set up Celery for a small demo project. At the end of this tutorial, you will be able to setup a Celery web console monitoring your tasks.

Python Projects for Kids - Book Reviewbook reviewThat's it :) ... Go read the review.

Python 2.7.12 releasedreleaseThe Python 2.7.x series has a new bugfix release, Python 2.7.12, available for download.

Djangorecipe: easy test coverage reportsdjangoCode coverage reports help you see which parts of your code are still untested. Yes, it doesn’t say anything about the quality of your tests, but at the least it tells you which parts of your code have absolute the worst kind of tests: those that are absent :-)

Your Django Story: Meet Anna MakarudzeinterviewAnna Makarudze lives in Harare, Zimbabwe, and was born and raised in Masvingo. She is an an ICT consultant as well as a Python/Django developer.

anymail/django-anymail: Django email backends and webhooks for Mailgun, Postmark, SendGrid, SparkPost and moredjangoAnymail integrates several transactional email service providers (ESPs) into Django, with a consistent API that lets you use ESP-added features without locking your code to a particular ESP. It currently fully supports Mailgun, Postmark, SendGrid, and SparkPost, and has limited support for Mandrill.

Seven article series dedicated on mining twitter datatwitterNote these articles were published a year back, currently trending on social media. Curating it because it's pretty interesting and fun.

Exploring the NFL Draft with Pythonsport analysisAfter reading a couple of posts by Michael Lopez about the NFL draft, I decided to recreate some of his analysis using Python (instead of R). Post being - https://statsbylopez.com/2016/05/02/the-nfl-draft-where-we-stand-in-2016/

Text Analysis, Markov Chains and Bible Quotes Generator.machine learningText analysis and patterns are very interesting topics for me. I think every author has their own unique, distinctive pattern. The way they build sentences and phrases, not only in terms of tone, but the subconscious choices of word pairings and sequences. How their sentences begin and end, and what they put in the middle; they leave their cerebral fingerprints on paper, blog posts or even FaceBook status updates. I think it is a very achievable, having enough base data, to identify the author Note - It's a trivial example but a good starting point for someone looking to learn/explore Markov chains.

Upcoming Conference / User Group Meet
Fox Valley Python User Group
STL Python
Python Meeting Düsseldorf
Sydney Python User Group
PyGotham 2016
EuroPython 2016
PyOhio 2016

GraphDash - 95 Stars, 7 ForkA web-based dashboard built on graphs and their metadata.
Daily-Reddit-Wallpaper - 77 Stars, 15 ForkChange your wallpaper to most upvoted image of the day on /r/wallpapers or from any other subreddit (on startup).
pipup - 19 Stars, 1 ForkBetter handling of Python pip's requirements.txt files
kiwi - 8 Stars, 1 ForkA simple DynamoDB ORM for python, based on boto2
pyscheduler - 8 Stars, 0 ForkCreate scheduled tasks at runtime easily (Django, Flask, Bottle etc.)
git_log_viewer - 7 Stars, 0 ForkGit log viewer
markovy - 6 Stars, 1 ForkPython module for Markov Chain algorithm
Categories: FLOSS Project Planets

Import Python: ImportPython Issue 78

Planet Python - Tue, 2016-10-25 03:39

Worthy Read
SQL Dashboards in a Flash
Periscope Data plugs directly into your database lets you run, save, and share analyses over billions of rows in seconds.Sponsor
Web Service Efficiency at Instagram with PythonperformanceInstagram currently features the world’s largest deployment of the Django web framework, which is written entirely in Python. We initially chose to use Python because of its reputation for simplicity and practicality, which aligns well with our philosophy of “do the simple thing first.” But simplicity can come with a tradeoff: efficiency. Instagram has doubled in size over the last two years and recently crossed 500 million users

Python watscode snippetA "wat" is what I call a snippet of code that demonstrates a counterintuitive edge case of a programming language. (The name comes from this excellent talk by Gary Bernhardt.) If you're not familiar with the language, you might conclude that it's poorly designed when you see a wat. Often, more context about the language design will make the wat seem reasonable, or at least justified.

Coconut - Simple, elegant, Pythonic functional programming.Coconut is a simple, elegant, Pythonic functional programming language that compiles to Python. Since all valid Python is valid Coconut, joining the over 30,000 people already using Coconut will only extend and enhance what you're already capable of in Python.

Running a Python 3 REST application with Statsd and Consul support in DockerRESTToday, I’m going to go over setting up a very simple way to spin up a Python Falcon REST service that reports to Statsd as well as registering as a service with Consul along with setting up health checks. I’ve borrowed some ideas/code from several places and changed as needed.

PigarA fantastic tool to generate requirements file for your Python project, and more than that.

Turn Python Errors into Awesome
Quickly pinpoint what’s broken and why. Get the context and insights to defeat all application errors. Check out the Python Integration of Rollbar https://rollbar.com/docs/notifier/pyrollbar/ . Easily integrates with Django, Pyramid, Flask, Bottle, Twisted.Sponsor
Django 1.10 beta 1 releaseddjango, releaseAs part of the Django 1.10 release process, today we've released Django 1.10 beta 1, a preview/testing package that represents the second stage in the 1.10 release cycle and an opportunity for you to try out the changes coming in Django 1.10.

I missed pycon, what is your favorite talk that you recommend I watch ? - Reddit DiscussionvideoDiscussion on Reddit on Pycon talk recommendations.

A high availability Django setup on the cheap - Roland van LaardjangoRoland build an educational website that needed to be high available on a tight budget. Nice write up on how to achieve it.

BeginnersGuide/NonProgrammers - Python WikiresourceIf you've never programmed before, the tutorials on this page are recommended for you; they don't assume that you have previous experience. If you have programming experience, also check out the BeginnersGuide/Programmers page. This section is hosted on the Python website and is well curated and maintained.

Curated list of 80 Python coursesresourceLearn Python online – A curated list of courses on Python

How To Serve Django Applications with Apache and mod_wsgi on Ubuntu 16.04 - DigitalOceaninstallationIn this guide, we will demonstrate how to install and configure Django in a Python virtual environment. We'll then set up Apache in front of our application so that it can handle client requests directly before passing requests that require application logic to the Django app. We will do this using the mod_wsgi Apache module that can communicate with Django over the WSGI interface specification.

An Interview with Pythonista Katharine JarmulinterviewThis interview with Pythonista Katharine Jarmul focuses on data work. A couple of events provide context. Katharine is presenting a talk titled "How Machine Learning Changed Sentiment Analysis, or I Hate You, Computer ????" at this year's Sentiment Analysis Symposium, July 12, 2016 in New York, following which she's offering a class, Learn Big Data Wrangling with Python, July 13-14, also in New York.

Simple type checked objects in Pythoncode snippetCode snippet to check types.

Upcoming Conference / User Group Meet
PyDelhi Meetup
Inland Empire Pyladies
Monthly Python user group in Sheffield, United Kingdomhttp://groups.google.com/group/python-sheffieldTwitter: @pysheff
Fox Valley Python User Group
Python Meeting Düsseldorf
Sydney Python User Group

coursera-dl-all - 99 Stars, 14 ForkExtend the Coursera Downloader by downloading quizzes and assignments (and hopefully forum posts soon!). Uses coursera-dl in the process.
reductio - 80 Stars, 2 Forkr.a.a. reduces all C programs to identical instruction streams; that is, the instructions executed by the processor become the same for every program. The demonstration uses C and x86, but is not unique to either of these - the concept is easily adapted to most languages and architectures.
socli - 64 Stars, 8 ForkStack overflow command line stack overflow client written in python. Using SoCLI you can search and browse stack overflow without leaving the terminal.
TensorflowKR-2016-talk-debugging - 55 Stars, 10 ForkSlides and supplementary codes for my talk 'Debugging Tips on Tensorflow' (2016) https://wookayin.github.io/TensorflowKR-2016-talk-debugging
emojibot - 8 Stars, 0 ForkThis is a bot that adds custom emojis your slack team; *inline in Slack*. No more putzing around in your web browser.
ank - 7 Stars, 0 ForkPython microservices for Queue, Streaming, REST-API and Schedule task.
Effective-Python-Penetration-Testing - 5 Stars, 2 ForkThis is the code repository for [Effective Python Penetration Testing], published by Packt Publishing. It contains all the required files to run the code. This book is ideal for those who are comfortable with Python or a similar language and need no help with basic programming concepts but want to understand the basics of penetration testing and the problems pentesters face.
gh-streak - 5 Stars, 1 ForkA Python library, CLI, and API for fetching GitHub contribution streaks.
PAN-Card-OCR - 2 Stars, 0 ForkRetrive meaningful information from PAN Card image using tesseract-ocr. Indian developer would find this interesting perhaps.
Categories: FLOSS Project Planets

Import Python: ImportPython Issue 77

Planet Python - Tue, 2016-10-25 03:39

Worthy Read
Have an idea for a Python App ?
Create a new Python app, free with Azure App Service. Curator's Note - Select Python from the language dropdown and select Python. You have templates for django, flask, bottle. Check it out.Sponsor
Python: How to Create an Exception Logging Decoratorcode snippetThe other day, I decided I wanted to create a decorator to catch exceptions and log them. I found a rather complex example on Github that I used for some ideas on how to approach this task and came up with the following.

Webcast: Doing Math with PythonwebcastJoin Amit Saha, author of Doing Math with Python, in this hands-on webcast, and learn how to use Python to solve calculus problems, make sense of numbers with graphs and statistics, do symbolic math with SymPy, and perform basic machine-learning tasks.

Custom python-social-auth PipelinevideoSometimes when you authenticate with a social service you need to do custom "stuff" in your code base beyond creating a user object. Learn how to get started simply adding a custom function do the python-social-auth pipeline.

PyBay - a San Francisco mini-PyCon in late AugustpyconPyBay is an intimate version of the highly successful PyCon conference. It's a weekend gathering for the Bay Area's Python community to learn, share, and hack together while having fun and taking care of our minds and bodies.

Turn Errors into Awesome
Quickly pinpoint what’s broken and why. Get the context and insights to defeat all application errors.Sponsor
Python 3.5.2rc1 and Python 3.4.5rc1 are now availablenew releasePython 3.5.2rc1 and Python 3.4.5rc1 are now available for download.

Python advisory: HTTP header injection in urllibsecurityPython's built-in URL library ("urllib2" in 2.x and "urllib" in 3.x) is vulnerable to protocol stream injection attacks (a.k.a. "smuggling" attacks) via the http scheme. If an attacker could convince a Python application using this library to fetch an arbitrary URL, or fetch a resource from a malicious web server, then these injections could allow for a great deal of access to certain internal services.

Get Django ReadydjangoThis Blog is meant for the learners who wish to get an understanding of the framework "Django" and familiarize with web development concepts . Good entry level tutorial for Django.

Your Django Story: Meet Kelsey Gilmore-InnisinterviewKelsey Gilmore-Innis is the CTO of Sexual Health Innovations, and a transplanted San Franciscan living in Portland. Somehow, despite growing up in San Francisco during the thick of the first tech boom and bust, I never was exposed to code until adulthood. I took a programming for non-majors course a few years into college on a bit of a whim and, besides enjoying the subject matter, saw a path to a career. It took a little more time, but I ended up graduating with a CS degree.

PyCon 2016 RecappyconusPyCon, beyond being the best community event for Python developers, is also an event that we happily began thinking about eleven months ago. Almost as soon as PyCon 2015 ended, we had the good fortune of planning the look and feel of PyCon 2016 with organizer extraordinaires Ewa Jodlowska, Diana Clark, and new this year, Brandon Rhodes. Our team has loved working with the organizers on the PyCon websites for the past three years now. They’re great people who always prioritize the needs of PyCon attendees, whether that’s babysitting services or a smooth PyCon web experience.

PyCon JP 2016 Call for ProposalscommunityPyCon JP is a conference where Python users, or people interested in Python, gather to learn from each other and meet other members of the community. We will hold PyCon Japan on September 20-24 this year in Tokyo and are looking for talk proposals.

The PSF's Growing SuccessPSFIn honor of the 2016-2017 board of director's first board meeting today, I wanted to share the PSF's growing success with the public!


Senior Python Software Engineer at dubizzleDubai - United Arab EmiratesWe are looking for a Senior Software Engineer in our great office at Dubai, UAE. Delivering A-class products that scale to millions of users, applying new industry trends and challenging your peers are all part of the daily gig.

Upcoming Conference / User Group Meet
PyCon Singapore 2016
EuroPython 2016
PyOhio 2016
PyCon Australia 2016
EuroScipy 2016
Kiwi PyCon

requests-respectful - 172 Stars, 1 ForkMinimalist Requests wrapper to work within rate limits of any amount of services simultaneously. Parallel processing friendly.
deep-rl-tensorflow - 144 Stars, 18 ForkTensorFlow implementation of Deep Reinforcement Learning papers
yolog - 97 Stars, 4 ForkBeautify your Git Logs !
tickets - 92 Stars, 27 Forktickets query via command line.
SpaceshipGenerator - 89 Stars, 7 ForkA Blender script to procedurally generate 3D spaceships
flickrup - 69 Stars, 5 ForkFlickr auto uploader
preflyt - 21 Stars, 1 ForkA lightweight application environment checker
contributors - 15 Stars, 0 ForkA command-line script to get all the contributors for one or more GitHub projects.
starbot - 12 Stars, 2 ForkGenerates N stars for any repo
Categories: FLOSS Project Planets

Jaldhar Vyas: Aaargh gcc 5.x You Suck

Planet Debian - Tue, 2016-10-25 02:45

Aaargh gcc 5.x You Suck

I had to write a quick program today which is going to be run many thousands of times a day so it has to run fast. I decided to do it in c++ instead of the usual perl or javascript because it seemed appropriate and I've been playing around a lot with c++ lately trying to update my knowledge of its' modern features. So 200 LOC later I was almost done so I ran the program through valgrind a good habit I've been trying to instill. That's when I got a reminder of why I avoid c++.

==37698== HEAP SUMMARY: ==37698== in use at exit: 72,704 bytes in 1 blocks ==37698== total heap usage: 5 allocs, 4 frees, 84,655 bytes allocated ==37698== ==37698== LEAK SUMMARY: ==37698== definitely lost: 0 bytes in 0 blocks ==37698== indirectly lost: 0 bytes in 0 blocks ==37698== possibly lost: 0 bytes in 0 blocks ==37698== still reachable: 72,704 bytes in 1 blocks ==37698== suppressed: 0 bytes in 0 blocks

One of things I've learnt which I've been trying to apply more rigorously is to avoid manual memory management (news/deletes.) as much as possible in favor of modern c++ features such as std::unique_ptr etc. By my estimation there should only be three places in my code where memory is allocated and none of them should leak. Where do the others come from? And why is there a missing free (or delete.) Now the good news is that valgrind is saying that the memory is not technically leaking. It is still reachable at exit but that's ok because the OS will reclaim it. But this program will run a lot and I think it could still lead to problems over time such as memory fragmentation so I wanted to understand what was going on. Not to mention the bad aesthetics of it.

My first assumption (one which has served me well over the years) was to assume that I had screwed up somewhere. Or perhaps it could some behind the scenes compiler magic. It turned out to be the latter -- sort of as I found out only after two hours of jiggling code in different ways and googling for clues. That's when I found this Stack Overflow question which suggests that it is either a valgrind or compiler bug. The answer specifically mentions gcc 5.1. I was using Ubuntu LTS which has gcc 5.4 so I have just gone ahead and assumed all 5.x versions of gcc have this problem. Sure enough, compiling the same program on Debian stable which has gcc 4.9 gave this...

==6045== ==6045== HEAP SUMMARY: ==6045== in use at exit: 0 bytes in 0 blocks ==6045== total heap usage: 3 allocs, 3 frees, 10,967 bytes allocated ==6045== ==6045== All heap blocks were freed -- no leaks are possible ==6045==

...Much better. The executable was substantially smaller too. The time was not a total loss however. I learned that valgrind is pronounced val-grinned (it's from Norse mythology.) not val-grind as I had thought. So I have that going for me which is nice.

Categories: FLOSS Project Planets

Russ Allbery: Review: Lord of Emperors

Planet Debian - Tue, 2016-10-25 00:04

Review: Lord of Emperors, by Guy Gavriel Kay

Series: Sarantine Mosaic #2 Publisher: Eos Copyright: 2000 Printing: February 2001 ISBN: 0-06-102002-8 Format: Mass market Pages: 560

Lord of Emperors is the second half of a work that began with Sailing to Sarantium and is best thought of as a single book split for publishing reasons. You want to read the two together and in order.

As is typical for this sort of two-part work, it's difficult to review the second half without spoilers. I'll be more vague about the plot and the characters than normal, and will mark one bit that's arguably a bit of a spoiler (although I don't think it would affect the enjoyment of the book).

At the end of Sailing to Sarantium, we left Crispin in the great city, oddly and surprisingly entangled with some frighteningly powerful people and some more mundane ones (insofar as anyone is mundane in a Guy Gavriel Kay novel, but more on that in a bit). The opening of Lord of Emperors takes a break from the city to introduce a new people, the Bassanids, and a new character, Rustem of Karakek. While Crispin is still the heart of this story, the thread that binds the entirety of the Sarantine Mosaic together, Rustem is the primary protagonist for much of this book. I had somehow forgotten him completely since my first read of this series many years ago. I have no idea how.

I mentioned in my review of the previous book that one of the joys of reading this series is competence porn: watching the work of someone who is extremely good at what they do, and experiencing vicariously some of the passion and satisfaction they have for their work. Kay's handling of Crispin's mosaics is still the highlight of the series for me, but Rustem's medical practice (and Strumosus, and the chariot races) comes close. Rustem is a brilliant doctor by the standards of the time, utterly frustrated with the incompetence of the Sarantine doctors, but also weaving his own culture's belief in omens and portents into his actions. He's more reserved, more laconic than Crispin, but is another character with focused expertise and a deep internal sense of honor, swept unexpectedly into broader affairs and attempting to navigate them by doing the right thing in each moment. Kay fills this book with people like that, and it's compelling reading.

Rustem's entrance into the city accidentally sets off a complex chain of events that draws together all of the major characters of Sailing to Sarantium and adds a few more. The stakes are no less than war and control of major empires, and here Kay departs firmly from recorded history into his own creation. I had mentioned in the previous review that Justinian and Theodora are the clear inspirations for this story; that remains true, and many other characters are easy to map, but don't expect history to go here the way that it did in our world. Kay's version diverges significantly, and dramatically.

But one of the things I love the most about this book is its focus on the individual acts of courage, empathy, and ethics of each of the characters, even when those acts do not change the course of empires. The palace intrigue happens, and is important, but the individual acts of Kay's large cast get just as much epic narrative attention even if they would never appear in a history book. The most globally significant moment of the book is not the most stirring; that happens slightly earlier, in a chariot race destined to be forgotten by history. And the most touching moment of the book is a moment of connection between two people who would never appear in history, over the life of a third, that matters so much to the reader only because of the careful attention to individual lives and personalities Kay has shown over the course of a hundreds of pages.

A minor spoiler follows in the next paragraph, although I don't think it affects the reading of the book.

One brilliant part of Kay's fiction is that he doesn't have many villains, and goes to some lengths to humanize the actions of nearly everyone in the book. But sometimes the author's deep dislike of one particular character shows through, and here it's Pertennius (the clear analogue of Procopius). In a way, one could say the entirety of the Sarantine Mosaic is a rebuttal of the Secret History. But I think Kay's contrast between Crispin's art (and Scortius's, and Strumosus's) and Pertennius's history has a deeper thematic goal. I came away from this book feeling like the Sarantine Mosaic as a whole stands in contrast to a traditional history, stands against a reduction of people to dates and wars and buildings and governments. Crispin's greatest work attempts to capture emotion, awe, and an inner life. The endlessly complex human relationships shown in this book running beneath the political events occasionally surface in dramatic upheavals, but in Kay's telling the ones that stay below the surface are just as important. And while much of the other art shown in this book differs from Crispin's in being inherently ephemeral, it shares that quality of being the art of life, of complexity, of people in dynamic, changing, situational understanding of the world, exercising competence in some area that may or may not be remembered.

Kay raises to the level of epic the bits of history that don't get recorded, and, in his grand and self-conscious fantasy epic style, encourages the reader to feel those just as deeply as the ones that will have later historical significance. The measure of people, their true inner selves, is often shown in moments that Pertennius would dismiss and consider unworthy of recording in his history.

End minor spoiler.

I think Lord of Emperors is the best part of the Sarantine Mosaic duology. It keeps the same deeply enjoyable view of people doing things they are extremely good at while correcting some of the structural issues in the previous book. Kay continues to use a large cast, and continues to cut between viewpoint characters to show each event from multiple angles, but he has a better grasp of timing and order here than in Sailing to Sarantium. I never got confused about the timeline, thanks in part to more frequent and more linear scene cuts. And Lord of Emperors passes, with flying colors, the hardest test of a novel with a huge number of viewpoint characters: when Kay cuts to a new viewpoint, my reaction is almost always "yes, I wanted to see what they were thinking!" and almost never "wait, no, go back!".

My other main complaint about Sailing to Sarantium was the treatment of women, specifically the irresistibility of female sexual allure. Kay thankfully tones that down a lot here. His treatment of women is still a bit odd — one notices that five women seem to all touch the lives of the same men, and little room is left for Platonic friendship between the genders — but they're somewhat less persistently sexualized. And the women get a great deal of agency in this book, and a great deal of narrative respect.

That said, Lord of Emperors is also emotionally brutal. It's beautifully done, and entirely appropriate to the story, and Kay does provide a denouement that takes away a bit of the sting. But it's still very hard to read in spots if you become as invested in the characters and in the world as I do. Kay is writing epic that borders on tragedy, and uses his full capabilities as a writer to make the reader feel it. I love it, but it's not a book that I want to read too often.

As with nearly all Kay, the Sarantine Mosaic as a whole is intentional, deliberate epic writing, wearing its technique on its sleeve and making no apologies. There is constant foreshadowing, constant attempts to draw larger conclusions or reveal great principles of human nature, and a very open, repeated stress on the greatness and importance of events while they're being described. This works for me, but it doesn't work for everyone. If it doesn't work for you, the Sarantine Mosaic is unlikely to change your mind. But if you're in the mood for that type of story, I think this is one of Kay's best, and Lord of Emperors is the best half of the book.

Rating: 10 out of 10

Categories: FLOSS Project Planets

Gunnar Wolf: On the results of vote "gr_private2"

Planet Debian - Mon, 2016-10-24 21:46

Given that I started the GR process, and that I called for discussion and votes, I feel somehow as my duty to also put a simple wrap-around to this process. Of course, I'll say many things already well-known to my fellow Debian people, but also non-debianers read this.

So, for further context, if you need to, please read my previous blog post, where I was about to send a call for votes. It summarizes the situation and proposals; you will find we had a nice set of messages in debian-vote@lists.debian.org during September; I have to thank all the involved parties, much specially to Ian Jackson, who spent a lot of energy summing up the situation and clarifying the different bits to everyone involved.

So, we held the vote; you can be interested in looking at the detailed vote statistics for the 235 correctly received votes, and most importantly, the results:

First of all, I'll say I'm actually surprised at the results, as I expected Ian's proposal (acknowledge difficulty; I actually voted this proposal as my top option) to win and mine (repeal previous GR) to be last; turns out, the winner option was Iain's (remain private). But all in all, I am happy with the results: As I said during the discussion, I was much disappointed with the results to the previous GR on this topic — And, yes, it seems the breaking point was when many people thought the privacy status of posted messages was in jeopardy; we cannot really compare what I would have liked to have in said vote if we had followed the strategy of leaving the original resolution text instead of replacing it, but I believe it would have passed. In fact, one more surprise of this iteration was that I expected Further Discussion to be ranked higher, somewhere between the three explicit options. I am happy, of course, we got such an overwhelming clarity of what does the project as a whole prefer.

And what was gained or lost with this whole excercise? Well, if nothing else, we gain to stop lying. For over ten years, we have had an accepted resolution binding us to release the messages sent to debian-private given such-and-such-conditions... But never got around to implement it. We now know that debian-private will remain private... But we should keep reminding ourselves to use the list as little as possible.

For a project such as Debian, which is often seen as a beacon of doing the right thing no matter what, I feel being explicit about not lying to ourselves of great importance. Yes, we have the principle of not hiding our problems, but it has long been argued that the use of this list is not hiding or problems. Private communication can happen whenever you have humans involved, even if administratively we tried to avoid it.

Any of the three running options could have won, and I'd be happy. My #1 didn't win, but my #2 did. And, I am sure, it's for the best of the project as a whole.

Categories: FLOSS Project Planets

Enzolutions: My second BADCamp

Planet Drupal - Mon, 2016-10-24 20:00

Last week I had the opportunity of participating in my second BADCamp as part of my tour Around the Drupal world in 140+ days.

My first assistance was in 2012, a lot have changed from that time to now. At that time I was only assistant and was impress for the quality of the event and how vibrant was the community in San Francisco Bay Area.

The event itself doesn't change too much, the same venue, but a little more disperse in my opinion.

What change was my participation with Jesus Manuel Olivas one of my business partners at weKnow. We lead a training about Drupal 8 module development

Slides: [http://weknowinc.com/talks/2016/badcamp-training/]

Also we have a session about how to learn about Drupal 8 via debugging

Slides: http://weknowinc.com/talks/2016/badcamp-learn-drupal-8-via-debugging

Airplane Distance (Kilometers) San Jose, Costa Rica → Panama, Panama → San Francisco, USA 5.880 Previously 109.137 Total 115.017 Walking Distance (steps) Bay Area 58.408 Previously 1.936.685 Total 1.995.093 Train Distance (Kilometers) Today 0 Previously 528 Total 528 Bus/Car Distance (Kilometers) Today 796 Previously 2.944 Total 3.740
Categories: FLOSS Project Planets

Palantir: Highlights From HighEdWeb 2016

Planet Drupal - Mon, 2016-10-24 16:23
Highlights From HighEdWeb 2016 brandt Mon, 10/24/2016 - 15:23 Allison Manley Oct 24, 2016

HighEdWeb proved to be a valuable experience (again!) and we will be back.

In this post we will cover...
  • What makes HighEdWeb special
  • A few things we learned
  • Some of our favorite sessions

We want to make your project a success.

Let's Chat.

Last week I returned from the HigherEd Web Conference in Memphis, TN. This is one of my favorite conferences because the community is fantastic, the energy is high, the events are well-attended and create an inclusive atmosphere, and it’s just well-run. It’s also a very heavy Twitter user group . . . attendees love to tweet everything early and often, and each session is given its own hashtag so people can follow along. They also have a coveted award called the “Red Stapler” given for best in each track, culminating in a Best Of Conference award.

As if that wasn’t enough, the keynote speakers each year are amazing! This year, Kimberly Bryant of www.blackgirlscode.com was the first keynote. The community was so supportive of her organization’s work to bring computer science and code to young girls of color that people were donating money on the spot as she spoke, and tweeting it around to encourage others to keep the donations coming.

The final keynote was LeVar Burton. As the son of an educator, a Star Trek: The Next Generation actor, and Reading Rainbow creator, Mr. Burton is perfectly positioned to speak on the intersection of technology and education. He was funny, engaging, and accommodating. Plus he made everyone cry when talking about the important teachers in his life, and asked us to take a moment to reflect on ours. And kudos to the Reading Rainbow social media team, who were impressively following along with his talk and engaging via Twitter with the audience in real time.

This conference is mostly focused on implementers. It’s a conference of editors, designers, developers, admissions, and IT staff. This is a passionate group of people who want to do better by their schools via the school’s online presence, and I’m continually impressed by how creative the teams get within their limited resources (people, time and money). The smaller schools have the ability to be more nimble around internal politics and the lack of resources than larger institutions, but even the larger schools are making amazing things happen online.

Though this isn’t something that Palantir works in directly, schools and libraries are really getting savvy about social media. There were sessions on how schools were using Pinterest, the success around ways to text students (including admissions acceptances!), Snapchat (again . . . including admissions acceptances!), and more.

Schools are really looking about how to personalize and individualize the experience for students from a visual perspective, understanding that the younger generations are used to a personalized experience via Netflix, Spotify, and Amazon where recommendations can be made based on interests. Schools are responding to this by designing more around creating a “choose your own adventure” feel either through filtering majors and interests, or through messaging and branding in addition to social media.

I also learned a lot about the concerns that schools have in the areas of accessibility, crisis control, and rebuilding trust among faculty and administrators when past web projects either haven’t gone well, or due to internal power struggles over which content gets priority.

Between representing all the factions of a large organization, juggling the needs of multiple audiences, and having to keep up with the constantly changing workspace online, these small teams of web professionals have a lot to handle! And this annual meeting allows them to collaborate, learn from each other, compare notes, and celebrate their hard work.

Some Favorite Sessions

In addition to the Red Stapler winners, there were a lot of great sessions. Videos will be up on the schedule page eventually, but below are links to slides to some noteworthy ones:

My colleague Joe Allen-Black and I were able to give our presentation “Project Management: The Musical!” to this crowd as the last session on Monday. We added one more song since the last conference, and thankfully it all went smoothly without a full rehearsal! Considering the amounts of tweets there were related to the session and feedback we received in person, the attendees seemed to enjoy our musical take on how to get a project moving successfully from start to finish.

I learn so much from this conference every year about what the higher education community’s needs are for their online presence, what they are doing to solve those problems, and how we at Palantir can assist. HEW hosts regional conferences all over the country throughout the year, leading up to the big annual conference next October in Hartford, CT. I don’t yet know what my calendar holds for next year, but I look forward to being able to attend and see the great community of HEW often in 2017.

Stay connected with the latest news on web strategy, design, and development.

Sign up for our newsletter.
Categories: FLOSS Project Planets

Chris Lamb: Concorde

Planet Debian - Mon, 2016-10-24 14:59

Today marks the 13th anniversary since the last passenger flight from New York arrived in the UK. Every seat was filled, a feat that had become increasingly rare for a plane that was a technological marvel but a commercial flop….

  • Only 20 aircraft were ever built despite 100 orders, most of them cancelled in the early 1970s.
  • Taxiing to the runway consumed 2 tons of fuel.
  • The white colour scheme was specified to reduce the outer temperature by about 10°C.
  • In a promotional deal with Pepsi, F-BTSD was temporarily painted blue. Due to the change of colour, Air France were advised to remain at Mach 2 for no more than 20 minutes at a time.
  • At supersonic speed the fuselage would heat up and expand by as much as 30cm. The most obvious manifestation of this was a gap that opened up on the flight deck between the flight engineer's console and the bulkhead. On some aircraft conducting a retiring supersonic flight, the flight engineers placed their caps in this expanded gap, permanently wedging the cap as it shrank again.
  • At Concorde's altitude a breach of cabin integrity would result in a loss of pressure so severe that passengers would quickly suffer from hypoxia despite application of emergency oxygen. Concorde was thus built with smaller windows to reduce the rate of loss in such a breach.
  • The high cruising altitude meant passengers received almost twice the amount of radiation as a conventional long-haul flight. To prevent excessive exposure, the flight deck comprised of a radiometer; if the radiation level became too high, pilots would descend below 45,000 feet.
  • BA's service had a greater number of passengers who booked a flight and then failed to appear than any other aircraft in their fleet.
  • Market research later in Concorde's life revealed that customers thought Concorde was more expensive than it actually was. Ticket prices were progressively raised to match these perceptions.
  • The fastest transatlantic airliner flight was from New York JFK to London Heathrow on 7 February 1996 by British Airways' G-BOAD in 2 hours, 52 minutes, 59 seconds from takeoff to touchdown. It was aided by a 175 mph tailwind.

See also: A Rocket to Nowhere.

Categories: FLOSS Project Planets

Reproducible builds folks: Reproducible Builds: week 78 in Stretch cycle

Planet Debian - Mon, 2016-10-24 12:10

What happened in the Reproducible Builds effort between Sunday October 16 and Saturday October 22 2016:

Media coverage Upcoming events buildinfo.debian.net

In order to build packages reproducibly, you not only need identical sources but also some external definition of the environment used for a particular build. This definition includes the inputs and the outputs and, in the Debian case, are available in a $package_$architecture_$version.buildinfo file.

We anticipate the next dpkg upload to sid will create .buildinfo files by default. Whilst it's clear that we also need to teach dak to deal with them (#763822) its not actually clear how to handle .buildinfo files after dak has processed them and how to make them available to the world.

To this end, Chris Lamb has started development on a proof-of-concept .buildinfo server to see what issues arise. Source

Reproducible work in other projects
  • Ximin Luo submitted a patch to GCC as a prerequisite for future patches to make debugging symbols reproducible.
Packages reviewed and fixed, and bugs filed Reviews of unreproducible packages

99 package reviews have been added, 3 have been updated and 6 have been removed in this week, adding to our knowledge about identified issues.

6 issue types have been added:

Weekly QA work

During of reproducibility testing, some FTBFS bugs have been detected and reported by:

  • Chris Lamb (23)
  • Daniel Reichelt (2)
  • Lucas Nussbaum (1)
  • Santiago Vila (18)
diffoscope development tests.reproducible-builds.org
  • h01ger increased the diskspace for reproducible content on Jenkins. Thanks to ProfitBricks.
  • Valerie Young supplied a patch to make Python SQL interface more SQLite/PostgresSQL agnostic.
  • lynxis worked hard to make LEDE and OpenWrt builds happen on two hosts.

Our poll to find a good time for an IRC meeting is still running until Tuesday, October 25st; please reply as soon as possible.

We need a logo! Some ideas and requirements for a Reproducible Builds logo have been documented in the wiki. Contributions very welcome, even if simply by forwarding this information.

This week's edition was written by Chris Lamb & Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

Categories: FLOSS Project Planets

Chromatic: Responsive Images in Drupal 8 Using "srcset"

Planet Drupal - Mon, 2016-10-24 11:26

There are a handful of tutorials out there that explain how to use Drupal 8's responsive image and breakpoint modules. However, from what I could find, none of them address how to instruct Drupal to output simple tags that leverage srcset instead of using the element. Using srcset can be preferable in many circumstances and as such, knowing how to use it in Drupal 8 will be good for most developers to understand. Since I was so confused on how to make this work, I thought it would be worth sharing.

Why srcset over ?

Others have done a much better job explaining the "why" of this approach, namely Chris Coyier and Jason Grigsby. Read those posts for the full picture (heh) but suffice to say, if you're only needing to change the resolution of an image at different viewport sizes/densities, you should use srcset instead of the element. If you need images that vary more specifically (i.e. different croppings, styles, etc.) then you should use the element. The reason srcset is preferred, is that it allows the browser to make informed decisions about which version of the image to serve, whereas the element explicitly tells the browser which images to use and when via media queries. Browsers know way more about our end users than we (developers) can ever know, so why not let them make the hard decisions?

In short:
  • Use srcset for different image sizes/densities across breakpoints.
  • Use when you need separate art direction/crops across breakpoints.
Convincing Drupal to Use srcset

Drupal's UI for responsive images is confusing and (maybe) rightfully so. Responsive images are confusing. As a rule, Drupal tries really hard to not make assumptions about what the user needs. While I understand why we (the Drupal community) built core this way, this can lead to interfaces that even seasoned professionals have a hard time understanding. In this case, it wasn't clear to me how (or if) I could instruct Drupal that I just wanted an element with srcset and sizes, not a full element. Each method I tried produced a element.

After some digging, I found the core issue where this development and discussion occurred. After skimming that thread and poking around the accepted patch, I knew that what I wanted was possible, but I still couldn't grok how to configure things in Drupal to get the desired output.

Further digging revealed that the default template responsible is called: responsive-image.html.twig. Here is the full file that ships with the Stable base theme:

{# /** * @file * Theme override of a responsive image. * * Available variables: * - sources: The attributes of the tags for this tag. * - img_element: The controlling image, with the fallback image in srcset. * - output_image_tag: Whether or not to output an tag instead of a * tag. * * @see template_preprocess() * @see template_preprocess_responsive_image() */ #} {% if output_image_tag %} {{ img_element }} {% else %} {% if sources %} {# Internet Explorer 9 doesn't recognise source elements that are wrapped in picture tags. See http://scottjehl.github.io/picturefill/#ie9 #} {% for source_attributes in sources %} {% endfor %} {% endif %} {# The controlling image, with the fallback image in srcset. #} {{ img_element }} {% endif %}

The key in this template is the output_image_tag boolean. When true, the template renders an tag instead of the tag. Inline documentation within the responsive_image module's template_preprocess_responsive_image() function explains that this output_image_tag variable is true when:

There is only one source tag with an empty media attribute. This means we can output an image tag with the srcset attribute instead of a picture tag.

In this context, Drupal is basically saying: if the user hasn't setup multiple breakpoints for this style AND there isn't any specific media query attached. Or in other words, the user doesn't want the theme to determine the images served but wants to put that onus on the browser via srcset.


If your use-case is to of the "resolution-switching" variety, here is how you get the desired output (no picture element) within Drupal:

  1. Skip creating specific breakpoints in your theme, they aren't needed with this approach.
  2. Setup your different image styles at admin/config/media/image-styles. Usually something like, Author Small, Author Medium and Author Large.
  3. Create a responsive image style at admin/config/media/responsive-image-style. Make sure the Responsive Image module is enabled first.
  4. Ensure "Responsive Image" is selected for the "Breakpoint group".
  5. Choose a suitable "Fallback image style". Click "Save". The following screen is where the secret sauce is.
  6. Under the "1x Viewport Sizing []" fieldset, Select "Select multiple image styles and use the sizes attribute."
  7. Select each of Image styles you'd like to use.
  8. Adjust the Sizes value as needed. The default here 100vw is hard-coded for a good reason, it's a pretty sane default and works well in most situations. Customize this is you want even finer control. More on Sizes.
  9. Adjust your content type (or other entity) to use your new responsive image style either by altering the display formatter or handling programmatically.
  10. Verify results!

For a thorough explanation of how all of the bits of the Responsive Image module work, see the documentation at admin/help/responsive_image with the _Help module enabled._

You should see something like this for the output of that image field now (formatted for readability):

We now have an element with a srcset attribute containing paths to each of our image styles, a sizes attribute and a fallback src attribute which points to our fallback image style.

Now our browser will automatically determine which image to serve up based on the client's viewport size, pixel density, etc. Also, if the client's browser doesn't yet support srcset, it will simply fall back to the value set in the src attribute.

Armed with this knowledge, you can configure your Drupal sites to leverage the power of srcset and allow the browsers of the world to do the hard work of determining which image style to serve to your users.

Categories: FLOSS Project Planets

Unimity Solutions Drupal Blog: Drupal Board Staff Mixer

Planet Drupal - Mon, 2016-10-24 07:28

During our Board Retreat, the Staff Board mixer gave us a chance to meet the staff members and the staff members to meet the Board. The staff's commitment and involvement to the project is amazing and definitely needs a pat on the back!

Categories: FLOSS Project Planets

OSTraining: Upload PDF, Excel, Word Files to Drupal Content

Planet Drupal - Mon, 2016-10-24 06:17

An OSTraining member asked how he could attach files such as PDFs, Excel Documents, Word Documents to his content. 

There are many ways to do this in Drupal, but the easiest approach is to use the "File" field.

  • Login to your Drupal site.
  • Go to Structure > Content types.
  • You either add the field to an existing content type, or you can create a new one for specifically adding the files:
Categories: FLOSS Project Planets

Kdenlive packages available for testing

Planet KDE - Mon, 2016-10-24 04:35

Getting the correct dependencies and up to date version of required libraries has always been a challenge for Kdenlive users. So we are pleased to announce that we now provide binary packages for download. These packages contain the latest Kdenlive development version (git master), as well as current development versions of MLT, Frei0r, OpenCV. The GPU movit library is not included at this stage. There might be some performance hit due to the nature of the formats, but these packages will be most helpful to debug and test the alpha/beta versions so that we can provide better releases. It will also help to identify issues linked to missing dependencies or outdated libraries on your system.

So if you are ready, you can download Kdenlive’s first AppImage here:
Then, simply make the file executable and run it! In a terminal:

  • chmod a+x kdenlive-16.12-alpha1-x86_64.AppImage
  • ./kdenlive-16.12-alpha1-x86_64.AppImage

We also provide a Snap package available for testing on Ubuntu distros:

  • sudo snap install --edge --force-dangerous --devmode kdenlive-devel
  • kdenlive-devel

The snap package is in the edge channel and requires “devmode” because of current limitations in the snap format.

These packages will not touch your system’s libraries, and can easily be removed. Please leave your feedback in the comments or on our mailing list(registration required) / forums.

We will provide several updates to this packages before the Kdenlive 16.12 release so that you can follow the development / bug fixes.

Categories: FLOSS Project Planets

Liip: Drupal SearchAPI and result grouping

Planet Drupal - Mon, 2016-10-24 03:17

In this blog post I will present how, in a recent e-Commerce project built on top of Drupal7 (the former version of the Drupal CMS), we make Drupal7, SearchAPI and Commerce play together to efficiently retrieve grouped results from Solr in SearchAPI, with no indexed data duplication.

We used the SearchAPI and the FacetAPI modules to build a search index for products, so far so good: available products and product-variations can be searched and filtered also by using a set of pre-defined facets. In a subsequent request, a new need arose from our project owner: provide a list of products where the results should include, in addition to the product details, a picture of one of the available product variations, while keep the ability to apply facets on products for the listing. Furthermore, the product variation picture displayed in the list must also match the filter applied by the user: this with the aim of not confusing users, and to provide a better user experience.

An example use case here is simple: allow users to get the list of available products and be able to filter them by the color/size/etc field of the available product variations, while displaying a picture of the available variations, and not a sample picture.

For the sake of simplicity and consistency with Drupal’s Commerce module terminology, I will use the term “Product” to refer to any product-variation, while the term “Model” will be used to refer to a product.

Solr Result Grouping

We decided to use Solr (the well-known, fast and efficient search engine built on top of the Apache Lucene library) as the backend of the eCommerce platform: the reason lies not only in its full-text search features, but also in the possibility to build a fast retrieval system for the huge number of products we were expecting to be available online.

To solve the request about the display of product models, facets and available products, I intended to use the feature offered by Solr called Result-Grouping as it seemed to be suitable for our case: Solr is able to return just a subset of results by grouping them given an “single value” field (previously indexed, of course). The Facets can then be configured to be computed from: the grouped set of results, the ungrouped items or just from the first result of each group.

Such handy feature of Solr can be used in combination with the SearchAPI module by installing the SearchAPI Grouping module. The module allows to return results grouped by a single-valued field, while keeping the building process of the facets on all the results matched by the query, this behavior is configurable.

That allowed us to:

  • group the available products by the referenced model and return just one model;
  • compute the attribute’s facets on the entire collection of available products;
  • reuse the data in the product index for multiple views based on different grouping settings.
Result Grouping in SearchAPI

Due to some limitations of the SearchAPI module and its query building components, such plan was not doable with the current configuration as it would require us to create a copy of the product index just to apply the specific Result Grouping feature for each view.

The reason is that the features implemented by the SearchAPI Grouping are implemented on top of the “Alterations and Processors” functions of SearchAPI. Those are a set of specific functions that can be configured and invoked both at indexing-time and at querying-time by the SearchAPI module. In particular Alterations allows to programmatically alter the contents sent to the underlying index, while the Processors code is executed when a search query is built, executed and the results returned.
Those functions can be defined and configured only per-index.

As visible in the following picture, the SearchAPI Grouping module configuration could be done solely in the Index configuration, but not per-query.

Image 1: SearchAPI configuration for the Grouping Processor.

As the SearchAPI Grouping module is implemented as a SearchAPI Processor (as it needs to be able to alter the query sent to Solr and to handle the returned results), it would force us to create a new index for each different configuration of the result grouping.

Such limitation requires to introduce a lot of (useless) data duplication in the index, with a consequent decrease of performance when products are saved and later indexed in multiple indexes.
In particular, the duplication is more evident as the changes performed by the Processor are merely an alteration of:

  1. the query sent to Solr;
  2. the handling of the raw data returned by Solr.

This shows that there would be no need to index multiple times the same data.

Since the the possibility to define per-query processor sounded really promising and such feature could be used extensively in the same project, a new module has been implemented and published on Drupal.org: the SearchAPI Extended Processors module. (thanks to SearchAPI’s maintainer, DrunkenMonkey, for the help and review :) ).

The Drupal SearchAPI Extended Processor

The new module allows to extend the standard SearchAPI behavior for Processors and lets admins configure the execution of SearchAPI Processors per query and not only per-index.

By using the new module, any index can now be used with multiple and different Processors configurations, no new indexes are needed, thus avoiding data duplication.

The new configuration is exposed, as visible in the following picture, while editing a SearchAPI view under “Advanced > Query options”.
The SearchAPI processors can be altered and re-defined for the given view, a checkbox allows to completely override the current index setting rather than providing additional processors.

Image 2: View’s “Query options” with the SearchAPI Extended Processors module.


Conclusion: the new SearchAPI Extended Processors module has now been used for a few months in a complex eCommerce project at Liip and allowed us to easily implement new search features without the need to create multiple and separated indexes.
We are able to index Products data in one single (and compact) Solr index, and use it with different grouping strategies to build both product listings, model listings and model-category navigation pages without duplicating any data.
Since all those listings leverages the Solr FilterQuery query parameter to filter the correct set of products to be displayed, Solr can make use of its internal set of caches and specifically the filterCache to speed up subsequent searches and facets. This aspect, in addition to the usage of only one index, allows caches to be shared among multiple listings, and that would not be possible if separate indexes were used.

For further information, questions or curiosity drop me a line, I will be happy to help you configuring Drupal SearchAPI and Solr for your needs.

Categories: FLOSS Project Planets

Russ Allbery: Review: The Design of Everyday Things

Planet Debian - Mon, 2016-10-24 00:17

Review: The Design of Everyday Things, by Don Norman

Publisher: Basic Books Copyright: 2013 ISBN: 0-465-05065-4 Format: Trade paperback Pages: 298

There are several editions of this book (the first under a different title, The Psychology of Everyday Things). This review is for the Revised and Expanded Edition, first published in 2013 and quite significantly revised compared to the original. I probably read at least some of the original for a class in human-computer interaction around 1994, but that was long enough ago that I didn't remember any of the details.

I'm not sure how much impact this book has had outside of the computer field, but The Design of Everyday Things is a foundational text of HCI (human-computer interaction) despite the fact that many of its examples and much of its analysis is not specific to computers. Norman's goal is clearly to write a book that's fundamental to the entire field of design; not having studied the field, I don't know if he succeeded, but the impact on computing was certainly immense. This is the sort of book that everyone ends up hearing about, if not necessarily reading, in college. I was looking forward to filling a gap in my general knowledge.

Having now read it cover-to-cover, would I recommend others invest the time? Maybe. But probably not.

There are several things this book does well. One of the most significant is that it builds a lexicon and a set of general principles that provide a way of talking about design issues. Lexicons are not the most compelling reading material (see also Design Patterns), but having a common language is useful. I still remember affordances from college (probably from this book or something else based on it). Norman also adds, and defines, signifiers, constraints, mappings, and feedback, and talks about the human process of building a conceptual model of the objects with which one is interacting.

Even more useful, at least in my opinion, is the discussion of human task-oriented behavior. The seven stages of action is a great systematic way of analyzing how humans perform tasks, where those actions can fail, and how designers can help minimize failure. One thing I particularly like about Norman's presentation here is the emphasis on the feedback cycle after performing a task, or a step in a task. That feedback, and what makes good or poor feedback, is (I think) an underappreciated part of design and something that too often goes missing. I thought Norman was a bit too dismissive of simple beeps as feedback (he thinks they don't carry enough information; while that's not wrong, I think they're far superior to no feedback at all), but the emphasis on this point was much appreciated.

Beyond these dry but useful intellectual frameworks, though, Norman seems to have a larger purpose in The Design of Everyday Things: making a passionate argument for the importance of design and for not tolerating poor design. This is where I think his book goes a bit off the rails.

I can appreciate the boosterism of someone who feels an aspect of creating products is underappreciated and underfunded. But Norman hammers on the unacceptability of bad design to the point of tedium, and seems remarkably intolerant of, and unwilling to confront, the reasons why products may be released with poor designs for their eventual users. Norman clearly wishes that we would all boycott products with poor designs and prize usability above most (all?) other factors in our decisions. Equally clearly, this is not happening, and Norman knows it. He even describes some of the reasons why not, most notably (and most difficultly) the fact that the purchasers of many products are not the eventual users. Stoves are largely sold to builders, not kitchen cooks. Light switches are laid out for the convenience of the electrician; here too, the motive for the builder to spend additional money on better lighting controls is unclear. So much business software is purchased by people who will never use it directly, and may have little or no contact with the people who do. These layers of economic separation result in deep disconnects of incentive structure between product manufacturers and eventual consumers.

Norman acknowledges this, writes about it at some length, and then seems to ignore the point entirely, returning to ranting about the deficiencies of obviously poor design and encouraging people to care more about design. This seems weirdly superficial in this foundational of a book. I came away half-convinced that these disconnects of incentive (and some related problems, such as the unwillingness to invest in proper field research or the elaborate, expensive, and lengthy design process Norman lays out as ideal) are the primary obstacle in the way of better-designed consumer goods. If that's the case, then this is one of the largest, if not the largest, obstacle in the way of doing good design, and I would have expected this foundational of a book to tackle it head-on and provide some guidance for how to fight back against this problem. But Norman largely doesn't.

There is some mention of this in the introduction. Apparently much of the discussion of the practical constraints on product design in the business world was added in this revised edition, and perhaps what I'm seeing is the limitations of attempting to revise an existing text. But that also implies that the original took an even harder line against poor design. Throughout, Norman is remarkably high-handed in his dismissal of bad design, focusing more on condemnation than on an investigation of why bad design might happen and what we, as readers, can learn from that process to avoid repeating it. Norman does provide extensive analysis of the design process and the psychology of human interaction, but still left me with the impression that he believes most design failures stem from laziness and stupidity. The negativity and frustration got a bit tedious by the middle of the book.

There's quite a lot here that someone working in design, particularly interface design, should be at least somewhat familiar with: affordances, signifiers, the importance of feedback, the psychological model of tasks and actions, and the classification of errors, just to name a few. However, I'm not sure this book is the best medium for learning those things. I found it a bit tedious, a bit too arrogant, and weirdly unconcerned with feasible solutions to the challenge of mismatched incentives. I also didn't learn that much from it; while the concepts here are quite important, most of them I'd picked up by osmosis from working in the computing field for twenty years.

In that way, The Design of Everyday Things reminded me a great deal of the Gang of Four's Design Patterns, even though it's a more readable book and less of an exercise in academic classification. The concepts presented are useful and important, but I'm not sure I can recommend the book as a book. It may be better to pick up the same concepts as you go, with the help of Internet searches and shorter essays.

Rating: 6 out of 10

Categories: FLOSS Project Planets
Syndicate content