FLOSS Project Planets

A. Jesse Jiryu Davis: Announcing Motor 1.2 Release Candidate, With MongoDB 3.6 Support

Planet Python - Wed, 2017-12-13 02:01

MongoDB 3.6 was released December 5. Today I’ve uploaded a release candidate for version 1.2 of Motor, the async Python driver for MongoDB. This will be a big release so I hope you try the release candidate and tell me if it works for you or if you find bugs.

Install the release candidate with pip:

python -m pip install motor==1.2rc0 Compatibility Changes
  • MongoDB: drop 2.4, add 3.6
  • Python: drop 2.6, continue to support 2.7 and 3.3+
  • Tornado: drop 3, continue to support Tornado 4.5+
  • aiohttp: support 2.0 and later, drop older version

See the Compatibility Matrix for the relationships among Motor, Python, Tornado, and MongoDB versions.

MongoDB 3.6 Features Change Streams

There’s a new method MotorCollection.watch to acquire a Change Stream on a collection:

async for change in collection.watch(): print(change)

I’ve written you a little sample app that watches a collection for changes and posts each notification over a websocket using the Tornado webserver:

Tornado Change Stream Example

Causal Consistency

There’s a new Session API to support causal consistency, meaning you can read your writes and perform monotonic reads, even reading from secondaries in a replica set or a sharded cluster. Create a session with MotorClient.start_session and use it for a sequence of related operations:

collection = client.db.collection with (await client.start_session()) as s: doc = {'_id': ObjectId(), 'x': 1} await collection.insert_one(doc, session=s) secondary = collection.with_options( read_preference=ReadPreference.SECONDARY) # Sessions are causally consistent by default, we can read the doc # we just inserted, even reading from a secondary. async for doc in secondary.find(session=s): print(doc) Array Filters

You can now update subdocuments in arrays within documents, and use “array filters” to choose which subdocuments to change. Pass the array_filters argument to MotorCollection.update_one, MotorCollection.update_many, MotorCollection.find_one_and_update, or MotorCollection.bulk_write.

For example if your document looks like this:

{ _id: 1, points: [ {x: 0, y: 0}, {x: 1, y: 0}, {x: 2, y: 0} ] }

You can update all subdocuments where x is greater than zero:

await collection.update_one( {'_id': 1}, {'$set': {'points.$[i].y': 5}}, array_filters=[{'i.x': {'$gt': 0}}]) “mongodb+srv://” URIs

This new URI scheme is convenient for short, maintainable URIs that represent large clusters, especially in Atlas. See PyMongo’s MongoClient for details.

Retryable Writes

Now, with a MongoDB 3.6 replica set or sharded cluster, Motor will safely retry any write that’s interrupted by a network glitch or a primary failover, without any risk of a double write. Just add retryWrites to the URI:


See PyMongo’s MongoClient for details about retryable writes.

Thanks to everyone who contributed to this version:

  • A. Jesse Jiryu Davis
  • Bernie Hackett
  • Jakub Wilk
  • Karol Horowski

A. Jesse Jiryu Davis

Categories: FLOSS Project Planets

Mike Driscoll: Flask 101: How to Add a Search Form

Planet Python - Wed, 2017-12-13 01:05

In our last article, we added a database to our Flask web application, but didn’t have a way to add anything to our database. We also didn’t have a way to view anything, so basically we ended up having a pretty useless web application. This article will take the time to teach you how to do the following:

  • Create a form to add data to our database
  • Use a form to edit data in our database
  • Create some kind of view of what’s in the database

Adding forms to Flask is pretty easy too, once you figure out what extension to install. I had heard good things about WTForms so I will be using that in this tutorial. To install WTForms you will need to install Flask-WTF. Installing Flask-WTF is pretty easy; just open up your terminal and activate the virtual environment we set up in our first tutorial. Then run the following command using pip:

pip install Flask-WTF

This will install WTForms and Flask-WTF (along with any dependencies) to your web app’s virtual environment.

Serving HTML Files

Originally when I started this series, all I was serving up on the index page of our web application was a string. We should probably spruce that up a bit and use an actual HTML file. Create a folder called “templates” inside the “musicdb” folder. Now create a file called “index.html” inside the “templates” folder and put the following contents in it:

<doctype html> <head> <title>Flask Music Database</title> </head>   <h2>Flask Music Database</h2>

Now before we update our web application code, let’s go ahead and create a search form for filtering our music database’s results.

Adding a Search Form

When working with a database, you will want a way to search for items in it. Fortunately creating a search form with WTForms is really easy. Create a Python script called “forms.py” and save it to the “musicdb” folder with the following contents:

# forms.py   from wtforms import Form, StringField, SelectField   class MusicSearchForm(Form): choices = [('Artist', 'Artist'), ('Album', 'Album'), ('Publisher', 'Publisher')] select = SelectField('Search for music:', choices=choices) search = StringField('')

Here we just import the items we need from the wtforms module and then we subclass the Form class. In our subclass, we create a selection field (a combobox) and a string field. This allows us to filter our search to the Artist, Album or Publisher categories and enter a string to search for.

Now we are ready to update our main application.

Updating the Main Application

Let’s rename our web application’s script from “test.py” to “main.py” and update it so it looks like this:

# main.py   from app import app from db_setup import init_db, db_session from forms import MusicSearchForm from flask import flash, render_template, request, redirect from models import Album   init_db()     @app.route('/', methods=['GET', 'POST']) def index(): search = MusicSearchForm(request.form) if request.method == 'POST': return search_results(search)   return render_template('index.html', form=search)     @app.route('/results') def search_results(search): results = [] search_string = search.data['search']   if search.data['search'] == '': qry = db_session.query(Album) results = qry.all()   if not results: flash('No results found!') return redirect('/') else: # display results return render_template('results.html', results=results)   if __name__ == '__main__': app.run()

We changed the index() function so it works with both POST and GET requests and told it to load our MusicSearchForm. You will note that when you first load the index page of your web app, it will execute a GET and the index() function will render our index.html that we just created. Of course, we didn’t actually add the form to our index.html yet, so that search form won’t appear yet.

There is also a search_results() that we added to handle really basic searches. However this function won’t be called until we actually implement a way to display the results. So let’s go ahead and make the search form visible to our users.

When I was learning how to create forms with wtforms, the Flask-WTF website recommended creating a template with a macro called “_formhelpers.html”. Go ahead and create a file of that name and save it to your “templates” folder. Then add the following to that file:

{% macro render_field(field) %} <dt>{{ field.label }} <dd>{{ field(**kwargs)|safe }} {% if field.errors %} <ul class=errors> {% for error in field.errors %} <li>{{ error }}</li> {% endfor %} </ul> {% endif %} </dd> {% endmacro %}

This syntax might look a little odd since it is obviously not just HTML. This is actually Jinja2 syntax, which is the templating language used by Flask. Basically anywhere you see the squiggly braces (i.e. {} ), you are seeing Jinja syntax. Here we pass in a field object and access its label and errors attributes. Feel free to lookup the documentation for additional information.

Now open up your “index.html” file and update it so that it has the following contents:

<doctype html> <head> <title>Flask Music Database</title> </head>   <h2>Flask Music Database</h2>   {% from "_formhelpers.html" import render_field %} <form method=post> <dl> {{ render_field(form.select) }} <p> {{ render_field(form.search) }} </dl> <p><input type=submit value=Search> </form>

The new code in this example shows how you can import the macro you created into your other HTML file. Next we set the form method to post and we pass the select widget and the search widget to our render_field macro. We also create a submit button with the following label: Search. When you press the Search button, it will post the data in the other two fields of the form to the page that it is on, which in this case is our index.html or “/”.

When that happens, the index() method in our main.py script will execute:

@app.route('/', methods=['GET', 'POST']) def index(): search = MusicSearchForm(request.form) if request.method == 'POST': return search_results(search)   return render_template('index.html', form=search)

You will note that we check which request method it is and if it’s the POST method, then we call the search_results() function. If you actually press the Search button at this stage, you will receive an Internal Server Error because we haven’t implemented “results.html” as of yet. Anyway, right now your web application should look something like this:

Let’s take a moment and get the results function doing something useful.

Updating the Results Functionality

Right now we don’t actually have any data in our database, so when we try to query it, we won’t get any results back. Thus we need to make our web application indicate that no results were found. To do that we need to update the “index.html” page:

<doctype html> <head> <title>Flask Music Database</title> </head>   <h2>Flask Music Database</h2>   {% with messages = get_flashed_messages() %} {% if messages %} <ul class=flashes> {% for message in messages %} <li>{{ message }}</li> {% endfor %} </ul> {% endif %} {% endwith %}   {% from "_formhelpers.html" import render_field %} <form method=post> <dl> {{ render_field(form.select) }} <p> {{ render_field(form.search) }} </dl> <p><input type=submit value=Search> </form>

You will note that the new code is a new block of Jinja. Here we grab “flashed” messages and display them. Now we just need to run the web application and try searching for something. If all goes as planned, you should see something like this when you do a search:

Wrapping Up

Now we have a neat little search form that we can use to search our database, although it frankly doesn’t really do all that much since our database is currently empty. In our next article we will focus on finally creating a way to add data to the database, display search results and edit the data too!

Download Code

Download a tarball of the code from this article: flask_musicdv_part_iii.tar

Other Articles in the Series Related Readings

Categories: FLOSS Project Planets

Electric Citizen: Twig for Drupal 8 Development: Twig Templating Part 1 of 2

Planet Drupal - Wed, 2017-12-13 00:17

Twig is a PHP templating engine and the default templating engine for Drupal 8. Part of the Symfony Framework, Twig syntax compiles templates down to plain PHP when rendered in the browser.

Twig offers advanced features like template inheritance, automatic escaping, variable filters and macros to improve development workflow. Drupal 8's Twig templates are "html.twig" templates, which means that you can mix Twig into HTML easier than you could mix PHP and HTML in Drupal 7’s PHP templates.

Syntax in Twig is somewhat different than PHP, with three main kinds of delimiters:

  • {{ }} prints the content of variables or expressions. This is the equivalent of PHP’s print or echo.
  • {% %} executes "if" statements, variable definitions or for loops.
  • {# #} adds template comments that are not rendered in the page.

A very basic Twig structure to print a message using a variable might look like:
{%  set message = ‘Welcome to  my great website‘ %}
{{ message }}

Categories: FLOSS Project Planets

Full Stack Python: How to Build Your First Slack Bot with Python

Planet Python - Wed, 2017-12-13 00:00

Bots are a useful way to interact with chat services such as Slack. If you have never built a bot before, this post provides an easy starter tutorial for combining the Slack API with Python to create your first bot.

We will walk through setting up your development environment, obtaining a Slack API bot token and coding our simple bot in Python.

Tools We Need

Our bot, which we will name "StarterBot", requires Python and the Slack API. To run our Python code we need:

It is also useful to have the Slack API docs handy while you're building this tutorial.

All the code for this tutorial is available open source under the MIT license in the slack-starterbot public repository.

Establishing Our Environment

We now know what tools we need for our project so let's get our development environment set up. Go to the terminal (or Command Prompt on Windows) and change into the directory where you want to store this project. Within that directory, create a new virtualenv to isolate our application dependencies from other Python projects.

virtualenv starterbot

Activate the virtualenv:

source starterbot/bin/activate

Your prompt should now look like the one in this screenshot.

The official slackclient API helper library built by Slack can send and receive messages from a Slack channel. Install the slackclient library with the pip command:

pip install slackclient

When pip is finished you should see output like this and you'll be back at the prompt.

We also need to create a Slack App to recieve an API token for your bot. Use "Starter Bot" as your App name. If you are signed into more than one workspace, pick a Development Workspace from the dropdown.

After submitting the form, keep the app configuration page open.

Slack APIs and App Configuration

We want our Starter Bot to appear like any other user in your team - it will participate in conversations inside channels, groups, and DMs. In a Slack App, this is called a bot user, which we set up by choosing "Bot Users" under the "Features" section. After clicking "Add a Bot User", you should choose a display name, choose a default username, and save your choices by clicking "Add Bot User". You'll end up with a page that looks like the following:

The slackclient library makes it simple to use Slack's RTM API and Web API. We'll use both to implement Starter Bot, and they each require authentication. Conveniently, the bot user we created earlier can be used to authenticate for both APIs.

Click on the "Install App" under the "Settings" section. The button on this page will install the App into our Development Workspace. Once the App is installed, it displays a bot user oauth access token for authentication as the bot user.

A common practice for Python developers is to export secret tokens as environment variables. Back in your terminal, export the Slack token with the name SLACK_BOT_TOKEN:

export SLACK_BOT_TOKEN='your bot user access token here'

Nice, now we are authorized to use the Slack RTM and Web APIs as a bot user.

Coding Our Starter Bot

We've got everything we need to write the Starter Bot code. Create a new file named starterbot.py and include the following code in it.

import os import time import re from slackclient import SlackClient

With our dependencies imported we can use them to obtain the environment variable values and then instantiate the Slack client.

# instantiate Slack client slack_client = SlackClient(os.environ.get('SLACK_BOT_TOKEN')) # starterbot's user ID in Slack: value is assigned after the bot starts up starterbot_id = None # constants RTM_READ_DELAY = 1 # 1 second delay between reading from RTM EXAMPLE_COMMAND = "do" MENTION_REGEX = "^<@(|[WU].+)>(.*)"

The code instantiates the SlackClient client with our SLACK_BOT_TOKEN exported as an environment variable. It also declares a variable we can use to store the Slack user ID of our Starter Bot. A few constants are also declared, and each of them will be explained as they are used in the code that follows.

if __name__ == "__main__": if slack_client.rtm_connect(with_team_state=False): print("Starter Bot connected and running!") # Read bot's user ID by calling Web API method `auth.test` starterbot_id = slack_client.api_call("auth.test")["user_id"] while True: command, channel = parse_bot_commands(slack_client.rtm_read()) if command: handle_command(command, channel) time.sleep(RTM_READ_DELAY) else: print("Connection failed. Exception traceback printed above.")

The Slack client connects to the Slack RTM API. Once it's connected, it calls a Web API method (auth.test) to find Starter Bot's user ID.

Each bot user has a user ID for each workspace the Slack App is installed within. Storing this user ID will help the program understand if someone has mentioned the bot in a message.

Next, the program enters an infinite loop, where each time the loop runs the client recieves any events that arrived from Slack's RTM API. Notice that before the loop ends, the program pauses for one second so that it doesn't loop too fast and waste your CPU time.

For each event that is read, the parse_bot_commands() function determines if the event contains a command for Starter Bot. If it does, then command will contain a value and the handle_command() function determines what to do with the command.

We've laid the groundwork for processing Slack events and calling Slack methods in the program. Next, add three new functions above the previous snippet to complete handling commands:

def parse_bot_commands(slack_events): """ Parses a list of events coming from the Slack RTM API to find bot commands. If a bot command is found, this function returns a tuple of command and channel. If its not found, then this function returns None, None. """ for event in slack_events: if event["type"] == "message" and not "subtype" in event: user_id, message = parse_direct_mention(event["text"]) if user_id == starterbot_id: return message, event["channel"] return None, None def parse_direct_mention(message_text): """ Finds a direct mention (a mention that is at the beginning) in message text and returns the user ID which was mentioned. If there is no direct mention, returns None """ matches = re.search(MENTION_REGEX, message_text) # the first group contains the username, the second group contains the remaining message return (matches.group(1), matches.group(2).strip()) if matches else (None, None) def handle_command(command, channel): """ Executes bot command if the command is known """ # Default response is help text for the user default_response = "Not sure what you mean. Try *{}*.".format(EXAMPLE_COMMAND) # Finds and executes the given command, filling in response response = None # This is where you start to implement more commands! if command.startswith(EXAMPLE_COMMAND): response = "Sure...write some more code then I can do that!" # Sends the response back to the channel slack_client.api_call( "chat.postMessage", channel=channel, text=response or default_response )

The parse_bot_commands() function takes events from Slack and determines if they are commands directed at Starter Bot. There are many event types that our bot will encounter, but to find commands we only want to consider message events. Message events also have subtypes, but the commands we want to find won't have any subtype defined. The function filters out uninteresting events by checking these properties. Now we know the event represents a message with some text, but we want to find out if Starter Bot is being mentioned in the text. The parse_direct_mention() function will figure out of the message text starts with a mention, and then we compare that to the user ID we stored earlier for Starter Bot. If they are the same, then we know this is a bot command, and return the command text with the channel ID.

The parse_direct_mentions() function uses a regular expression to determine if a user is being mentioned at the beginning of the message. It returns the user ID and the remaining message (and None, None if no mention was found).

The last function, handle_command() is where in the future you'll add all the interesting commands, humor, and personality for Starter Bot. For now, it has just one example command: do. If the command starts with a known command, it will have an appropriate response. If not, a default response is used. The response is sent back to Slack by calling the chat.postMessage Web API method with the channel.

Here is how the entire program should look when it's all put together (you can also view the file in GitHub):

import os import time import re from slackclient import SlackClient # instantiate Slack client slack_client = SlackClient(os.environ.get('SLACK_BOT_TOKEN')) # starterbot's user ID in Slack: value is assigned after the bot starts up starterbot_id = None # constants RTM_READ_DELAY = 1 # 1 second delay between reading from RTM EXAMPLE_COMMAND = "do" MENTION_REGEX = "^<@(|[WU].+)>(.*)" def parse_bot_commands(slack_events): """ Parses a list of events coming from the Slack RTM API to find bot commands. If a bot command is found, this function returns a tuple of command and channel. If its not found, then this function returns None, None. """ for event in slack_events: if event["type"] == "message" and not "subtype" in event: user_id, message = parse_direct_mention(event["text"]) if user_id == starterbot_id: return message, event["channel"] return None, None def parse_direct_mention(message_text): """ Finds a direct mention (a mention that is at the beginning) in message text and returns the user ID which was mentioned. If there is no direct mention, returns None """ matches = re.search(MENTION_REGEX, message_text) # the first group contains the username, the second group contains the remaining message return (matches.group(1), matches.group(2).strip()) if matches else (None, None) def handle_command(command, channel): """ Executes bot command if the command is known """ # Default response is help text for the user default_response = "Not sure what you mean. Try *{}*.".format(EXAMPLE_COMMAND) # Finds and executes the given command, filling in response response = None # This is where you start to implement more commands! if command.startswith(EXAMPLE_COMMAND): response = "Sure...write some more code then I can do that!" # Sends the response back to the channel slack_client.api_call( "chat.postMessage", channel=channel, text=response or default_response ) if __name__ == "__main__": if slack_client.rtm_connect(with_team_state=False): print("Starter Bot connected and running!") # Read bot's user ID by calling Web API method `auth.test` starterbot_id = slack_client.api_call("auth.test")["user_id"] while True: command, channel = parse_bot_commands(slack_client.rtm_read()) if command: handle_command(command, channel) time.sleep(RTM_READ_DELAY) else: print("Connection failed. Exception traceback printed above.")

Now that all of our code is in place we can run our Starter Bot on the command line with the python starterbot.py command.

In Slack, create a new channel and invite Starter Bot or invite it to an existing channel.

Now start giving Starter Bot commands in your channel.

Wrapping Up

Alright, now you've got a simple Starter Bot with a bunch of places in the code you can add whatever features you want to build.

There is a whole lot more that could be done using the Slack RTM API and Python. Check out these posts to learn what you could do:

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

Categories: FLOSS Project Planets

Bryan Pendleton: A winter adventure in Zion

Planet Apache - Tue, 2017-12-12 22:19

It was time to go, so we got up and went. We packed our bags, with as much warm clothing as we could reasonably carry, flew to Las Vegas, picked up a nice new rental car (what a nice car the new Toyota Camry is!), and drove northeast on interstate 15.

About 45 minutes out of Vegas, we took a back road recommended by my wife's colleague, which took us about 15 miles off the freeway, into a Nevada State Park named Valley of Fire. After a short stop at the Visitor Center to get our bearings, we found the picnic area named Mouse's Tank, populated by the boldest little ground squirrels you could imagine, practicing their cutest poses to try to convince us to donate some of our lunch to them.

After lunch, we took a short walk to admire the Valley of Fire petroglyphs, which are nothing short of astounding.

Then we were back in the car again, and soon back on I-15, and not long after that we were through Nevada, and had sliced a corner off of Arizona, and were solidly into Utah, before we left the freeway to take Utah State Route 9 east into Zion National Park.

The days are short, this time of year, so even though we got to the park gates at about 5:45 PM, it was already pitch dark, and we crept along the park road quite slowly, peeking around every corner for deer, trying to figure out where our turn was. Zion Lodge is located most of the way up the canyon road, deep in the main canyon, enjoying a location that can stand toe-to-toe with any hotel on the planet for claim to "Most Beautiful Lodge Location".

But, as I say, it was completely dark out, and we were exhausted, so we simply checked into our room (which was wonderful: spacious and elegant), had dinner at the lodge restaurant, and collapsed into bed.

Deep in the main canyon, sunset comes early and sunrise late, particularly this time of year. But up we got, the next morning, and bravely we set out to explore Zion National Park. Lo and behold, as the sun started to crawl slowly down the western walls of the canyon toward the valley floor, we found ourselves nearly alone in a place of tremendous beauty, with nearly as many mule deer as human visitors keeping us company on our explorations.

At the very end of the canyon road, one of the most famous trails is the Riverside Walk, which leads into the section of the Virgin River canyon known as The Narrows, launching spot for those interested in the sport of Canyoneering. We could barely imagine this, for at the time we walked the trail the temperature was 34 degrees, and a steady breeze was blowing, so we were fully encased in every shred of clothing we could layer upon ourselves, but at the trail's end there were nearly a dozen people, of all ages, clad in little more than long-sleeved swimsuits, waterproof hiking boots, and gaiters, setting off confidently into the rapidly-flowing, near-freezing waters of the Virgin River, headed upstream for adventure.

We had decided to work our way, slowly, back down the main canyon, and so we did, stopping to hike the Weeping Rock trail, the Emerald Pools trail, and the Watchman trail, among others, as well as stopping along the road for half an hour or so to watch people hiking up Walter's Wiggles (as well as rock climbing the cliff face below the Angels Landing trail).

No, we didn't do the Angels Landing trail. Yes, it's true: I ruled it out from the start. Uh, here's the reason why.

By the end of our first day, we were well and thoroughly exhausted, but also extremely pleased with the day.

There's just nothing like the experience of spending an entire day in a National Park: waking up in the park, spending all day in and around the park, and then remaining in the park when all the daily visitors go home, and it's just you lucky few. And the mule deer.

Once again we woke up the next morning in complete darkness, and made our way over to the lodge for breakfast, with aching muscles yet still aching for more.

Zion National Park is fairly large, even though compared to some national parks it's not gigantic, and I was hungry to see as much of the park as I could.

So we popped into the car and drove up Kolob Terrace Road, which in barely a dozen miles took us up from the 3,500 foot river elevation to the 7,000 foot elevation of Upper Kolob Terrace.

We were well-prepared: we had brought our lunch, and, as it turns out, we had brought the right clothing, for by the time we reached the Northgate Peaks trail it was already in the low 50's, and by the time we reached trail's end it was in the low 60's. Sunny skies, perfect temperatures, no bugs, and a nearly-level 2 mile hike to an amazing canyon viewpoint: is there any better way to spend a day in the mountains?

On our way back down, we stopped at Hoodoo City and tried to follow the trail over to see the peculiar rock formations, but it was slow, sandy going, and the closer we got to the rocks, the more they seemed to fade into the distance. Our decision was made for us when we met a couple returning from the trail who told us they were pretty sure they'd heard a mountain lion growling just a few dozen yards from the trail.

So back down the hill we went, and decided to settle for a yummy dinner at the local brewpub.

All good things must come to an end, and it was time to return to civilization, so we got a good early start on our final day in the mountains and made a short stop at the third part of Zion National Park which is easily accessible: Kolob Canyons. Happily, we had just enough time to drive up to the end of the road to take in the truly remarkable views. The views from the roadside parking lot are superb; the views from the end of the Timber Creek Overlook trail are even better.

Back down I-15 to Las Vegas we went. My mother, who knows a lot about this part of the world, swears that U.S. 395 along the Eastern Sierra is the most beautiful road in the 48 states, and she's got a fine case, but I think that the stretch of I-15 from Las Vegas, Nevada to Cedar City, Utah is a serious contender, particularly on a clear winter's day when the view goes on forever (well, at least 50 miles).

It was as nice a way as one could ask to end as nice a weekend as one could hope for.

If you ever get a chance to visit Zion National Park in winter, take it.

Categories: FLOSS Project Planets

Wingware News: Wing Python IDE 6.0.9: December 13, 2017

Planet Python - Tue, 2017-12-12 20:00
This release adds support for Vagrant containers, improves support for Django and Plone/Zope, further improves remote development, fixes startup problems seen on some OS X systems, and makes about 35 other improvements.
Categories: FLOSS Project Planets

Justin Mason: Links for 2017-12-12

Planet Apache - Tue, 2017-12-12 18:58
  • The Case for Learned Index Structures

    ‘Indexes are models: a B-Tree-Index can be seen as a model to map a key to the position of a record within a sorted array, a Hash-Index as a model to map a key to a position of a record within an unsorted array, and a BitMap-Index as a model to indicate if a data record exists or not. In this exploratory research paper, we start from this premise and posit that all existing index structures can be replaced with other types of models, including deep-learning models, which we term learned indexes. The key idea is that a model can learn the sort order or structure of lookup keys and use this signal to effectively predict the position or existence of records. We theoretically analyze under which conditions learned indexes outperform traditional index structures and describe the main challenges in designing learned index structures. Our initial results show, that by using neural nets we are able to outperform cache-optimized B-Trees by up to 70% in speed while saving an order-of-magnitude in memory over several real-world data sets. More importantly though, we believe that the idea of replacing core components of a data management system through learned models has far reaching implications for future systems designs and that this work just provides a glimpse of what might be possible.’ Excellent follow-up thread from Henry Robinson: https://threadreaderapp.com/thread/940344992723120128 ‘The fact that the learned representation is more compact is very neat. But also it’s not really a surprise that, given the entire dataset, we can construct a more compact function than a B-tree which is *designed* to support efficient updates.’ […] ‘given that the model performs best when trained on the whole data set – I strongly doubt B-trees are the best we can do with the current state-of-the art.’

    (tags: data-structures ml google b-trees storage indexes deep-learning henry-robinson)

  • Internet protocols are changing

    per @mnot. HTTP/2; TLS 1.3; QUIC and UDP; and DOH (DNS over HTTP!)

    (tags: crypto encryption http https protocols http2 tls quic udp tcp dns tunnelling)

Categories: FLOSS Project Planets

Jacob Rockowitz: Who is creating add-ons to the Webform module?

Planet Drupal - Tue, 2017-12-12 17:51

As the Webform module progresses towards a stable release and developers are getting familiar with Drupal 8, some developers are starting to contribute some very cool add-ons to the Webform module’s ecosystem.

Drupal is about collaboratively creating robust, stable, and extendable APIs that developers can use to improve and customize Drupal's functionality. Drupal's core Form API (FAPI) is the backbone of the Webform module for Drupal 8. The Webform module would not exist without FAPI, so here is a shout out to all the past maintainers of Drupal FAPI and its current maintainers, Alex Bronstein (effulgentsia) and Tim Plunkett (tim.plunkett).

Because the Webform module Drupal 8 is an entirely new code base, the entire Webform related project ecosystem must be rebuilt from scratch. I see this as a challenge and an opportunity to rethink the functionality that the core Webform module provides and how it is extended.

I strongly feel that the Webform module must be a complete form building solution, which inspires the Drupal community to do what it does best, extend the heck out of powerful APIs

To encourage people to extend the Webform module I decided to track contributed modules within the Webform UI and on Drupal.org in a section that I am calling "Add-Ons."

Laurent is making it possible to analyze webform submissions

Laurent BARAN (lbaran)

Analysis and charts are one of the few features that were available in Webform...Read More

Categories: FLOSS Project Planets

Drupal.org blog: What's new on Drupal.org - November 2017

Planet Drupal - Tue, 2017-12-12 16:40

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Announcements Clarifications to Drupal licensing policy

There have been some long-standing questions about Drupal project licensing policy. In collaboration with the Licensing Working Group and Dries, we have updated the official licensing policy with the following clarifications and changes:

  • We explicitly affirm that code with a GPL-compatible license can be included in a Drupal.org hosted project, but will be redistributed under our standard "GPL2 or later" policy.
  • We clarify that GPL-incompatible non-code assets may be packaged and/or distributed "in aggregate" with GPL code in Drupal.org projects, per the final stipulation of Sec 3-2 of the GPL, so long as the maintainer has the rights to do so.
  • We clarify that Drupal.org hosted projects can depend on and/or link to GPL-incompatible code (via composer, for example), but that Drupal.org cannot host or distribute those GPL-incompatible dependencies.
  • We explicitly affirm our interpretation of the GPL that a Drupal service provider's act of assembling a codebase while under contract to a client does not fall under the more restrictive terms of 'distribution' per the GPL.

You can find more detail about these changes in the Repository Usage Policy and Licensing FAQ.

Welcoming new staff members

We want to welcome two new members to the Drupal Association team. Brooke Candelaria who will be taking over as Conference Director, joins us from Houston, TX. Brooke has a background in PR and event management, and has worked with other open source communities in the past, for example working on LinuxWorld and, most recently, with the Python community. Brooke's focus will be on evolving DrupalCon to meet the needs of every persona within our community, and helping to make the Con more sustainable and scalable.

Rachel Lawson is taking on the role of Community Liaison. Based in the UK, Rachel has been a tremendous member of the community in her own right, participating in Camps, helping to organize Mentored Sprints, and serving on the Community Working Group. Rachel will be helping the community to understand the role and the functioning of the Drupal Association, while also keeping the Drupal Association more closely tied to all parts of our diverse community. We hope you'll welcome them!

DrupalCon Updates Launched new DrupalCon brand

We've just launched a new unified look and feel for DrupalCon, that will carry into all of our events moving forward. This will help give DrupalCon a stronger identity among OSS events, and also give us a place to put centralized content that applies to DrupalCon as a whole, rather than just a singular event.

Launched DrupalCon Nashville site

Alongside our new brand, we've launched the DrupalCon Nashville website, the first of the events to use the new unified look and feel. Registration is open now, as well as the call for papers. Take a look at some of the new tracks! \

We'll see you there!

Sponsorship makes DrupalCon possible - learn why you should be a part of it.

Drupal.org Updates Reminder: New issue shortcuts and friendly url structure

Last month we announced that we would be implementing some changes to the issue url structure, as well as some shortcuts to help users navigate to issues more easily on Drupal.org. These changes went live in November. Here's a primer:

URL Pattern for issues:

https://drupal.org/project/drupal/issues/2922626 When an issue is moved between projects the alias will be updated.


The search bar will now automatically redirect you to a node if you enter its id directly: A new menu callback will help you get to issues with a shorter url string: https://drupal.org/i/

New individual member directory

We've been overhauling the Drupal Association membership experience, and as part of that process we now have a new directory of Drupal Association members. This new directory also includes new filters and sorting options so that you can see the latest community members to join the Association and filter by country, username, or first and last name.

In development: new organization member directory

We're working on a new directory of organization members as well. The current directory allows you to filter by organization type, and we will be adding a filter for Association membership as well. If you have feedback on this directory, please let us know!


As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who make it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association. Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Categories: FLOSS Project Planets

Free software needs net neutrality! This is our LAST CHANCE to save it

FSF Blogs - Tue, 2017-12-12 16:20

We have two more days to do everything we can to make our voices heard on this monumental issue. Below we have a sample script for calling the United States Congress, ideas for social media posts, and a bit about why free software needs net neutrality. If you want to read even more about why the Free Software Foundation (FSF) loves net neutrality, you can view this post on our blog.

Net Neutrality protest at the Boylston Verizon Store in Boston, MA on December 7th, 2017. Photo by Ruben Rodriguez. Creative Commons Attribution-ShareAlike 4.0

Call, call, call

If you are in the US, call Congress today. Nervous? Try using the following script:

Hello, I live in CITY/STATE. I am calling to urge you to support net neutrality and stop the FCC from removing common carrier status from Internet Service Providers like Comcast and Verizon. This is the only thing we have protecting a free Internet, which everyone needs. Thank you for your time.

Don't know who to call?

  • You can find your Representative and call them.

  • Dial the House of Representatives at (202) 224-3121 and they will connect you.

  • Call your Senator directly. Find their number here.

  • Dial the Senate at (202) 224-3121 and they will connect you.

(Note: The number for the House and the Senate is a switchboard that will direct your call.)

Post and share

No matter where you live, if you use social media, you can join others in making as much noise as you can in support of net neutrality. Share your favorite articles, change your profile photos, tell people you care about a free Web, and that, today, net neutrality is the way to maintain a free Web in the United States.

Need a sample message?

  • I support #netneutrality and #freesoftware!

  • #Freesoftware needs #netneutrality

  • I need #netneutrality because I need a free Web

You can also share this post, or look though our social media history for more ideas.

Why net neutrality?

There are so many reasons why we think net neutrality is important--and why it's necessary for free software. We'll briefly mention that:

  • Free software is built collaboratively by using Web tools and Internet connections around the world.

  • Free software is most easily discoverable thanks to pages like the Free Software Directory.

  • Need to update your system or software quickly? You need an Internet connection to make that happen.

  • Organizations like the FSF use the Web to educate and share free software ideology and tools.

  • The FSF itself is run by a small team spread across the globe. Every day we use tools like IRC to communicate and work for user freedom together, with one another, and with you.

  • We promote decentralized free software replacements for centralized software services, and that losing net neutrality means the centralized services will have a huge built-in advantage.

Without a free Web and free Internet, what we--this includes you--can do online will be limited by what ISPs like Comcast and Verizon want you to be able to do. They will have the legal right to control which Web sites we can access and how fast that access will be--and they will take advantage of their new ability to extort even greater fees from Web site and consumers alike.

We're asking you to take the time today and tomorrow to call Congress. Make some noise online. Tell your friends. Save the Internet.

Fellow digital rights organizations are running a campaign called Break the Internet. While we would love to share these other ways to take action, we cannot in good conscience link to the relevant Web pages because they require non-free Javascript.
Categories: FLOSS Project Planets

Freelock : A slick migration trick - convert columns to multi-value field with subfields

Planet Drupal - Tue, 2017-12-12 14:37

In the previous post on A custom quantity price discount for Drupal Commerce we created a compound field for price breaks, which was composed by two subfields. One for a quantity threshold, the other for the price at that threshold.

That post covered everything needed to get quantity discounts working within Drupal Commerce, but for this particular project, we also had to find a way to populate these price breaks through the integration with their Sage accounting system.

Drupal 8Drupal MigrationDrupal PlanetIntegrationERP
Categories: FLOSS Project Planets

Python Anywhere: Introducing the beta for always-on tasks

Planet Python - Tue, 2017-12-12 13:47
.jab-post img { border: 2px solid #eeeeee; padding: 5px; }

Today we're starting the beta for a new paid feature on PythonAnywhere: always-on tasks. If you have a paid account and would like to try them out, drop us a line at support@pythonanywhere.com.

With always-on tasks, you can keep a script constantly running. Once the feature's been switched on for your account, they can be set up in the "Tasks" tab -- you specify a bash command (eg. "python3.6 /home/username/myscript.py"), and the system will start running it. If it exits -- say, if it crashes, or if the server has a problem -- it will be restarted quickly.

We'd be really grateful for any and all feedback people have about this. We've been testing it internally for some months, and a private beta has been running for a while -- so we're reasonably certain it's solid and reliable, but please do remember that it is currently beta, and there may be bugs we don't know about.

Categories: FLOSS Project Planets

Continuum Analytics Blog: Parallel Python with Numba and ParallelAccelerator

Planet Python - Tue, 2017-12-12 13:28
With CPU core counts on the rise, Python developers and data scientists often struggle to take advantage of all of the computing power available to them. CPUs with 20 or more cores are now available, and at the extreme end, the Intel® Xeon Phi™ has 68 cores with 4-way Hyper-Threading. (That’s 272 active threads!) To …
Read more →
Categories: FLOSS Project Planets

Python Engineering at Microsoft: North Bay Python 2017 Recap

Planet Python - Tue, 2017-12-12 13:00

Last week I had the privilege to attend the inaugural North Bay Python conference, held in Petaluma, California in the USA. Being part of any community-run conference is always enjoyable, and to help launch a new one was very exciting. In this post, I'm going to briefly tell you about the conference and help you find recordings of some of the best sessions (and also the session that I presented).

Petaluma is a small city in Sonoma County, about one hour north of San Francisco. Known for their food and wine, it was a surprising location to find a conference, including for many locals who got to attend their first Python event.

If the photo to the right looks familiar, you probably remember it as the default Windows XP background image. It was taken in the area, inspired the North Bay Python logo, and doesn't actually look all that different from the hills surrounding Petaluma today.

Nearly 250 attendees converged on a beautiful old theatre to hear from twenty-two speakers. Topics ranged from serious topics of web application accessibility, inclusiveness, through to lighthearted talks on machine learning and Django, and the absolutely hilarious process of implementing merge sort using the import statement. All the videos can be found on the North Bay Python YouTube channel.

Recently I have been spending some of my time working on a proposal to add security enhancements to Python, similar to those already in Powershell. While Microsoft is known for being highly invested in security, not everyone shares the paranoia. I used my twenty-five minute session to raise awareness of how modern malware attacks play out, and to show how PEP 551 can enable security teams to better defend their networks.

(Image credit: VM Brasseur, CC-BY 2.0)

While I have a general policy of not uploading my presentation (slides are for speaking, not reading), here are the important links and content that you may be interested in:

Overall, the conference was a fantastic success. Many thanks to the organizing committee, Software Freedom Conservancy, and the sponsors who made it possible, and I am looking forward to attending in 2018.

Until the next North Bay Python though, we would love to have a chance to meet you at the events that you are at. Let us know in the comments what your favorite Python events are and the ones you would most like to have people from our Python team come and speak at.

Categories: FLOSS Project Planets

Keith Packard: Altos1.8.3

Planet Debian - Tue, 2017-12-12 12:44
AltOS 1.8.3 — TeleMega version 3.0 support and bug fixes

Bdale and I are pleased to announce the release of AltOS version 1.8.3.

AltOS is the core of the software for all of the Altus Metrum products. It consists of firmware for our cc1111, STM32L151, STMF042, LPC11U14 and ATtiny85 based electronics and Java-based ground station software.

This is a minor release of AltOS, including support for our new TeleMega v3.0 board and a selection of bug fixes

Announcing TeleMega v3.0

TeleMega is our top of the line flight computer with 9-axis IMU, 6 pyro channels, uBlox Max 7Q GPS and 40mW telemetry system. Version 3.0 is feature compatible with version 2.0, incorporating a new higher-perfomance 9-axis IMU in place of the former 6-axis IMU and separate 3-axis magnetometer.

AltOS 1.8.3

In addition to support for TeleMega v3.0 boards, AltOS 1.8.3 contains some important bug fixes for all flight computers. Users are advised to upgrade their devices.

  • Ground testing EasyMega and TeleMega additional pyro channels could result in a sticky 'fired' status which would prevent these channels from firing on future flights.

  • Corrupted flight log records could prevent future flights from capturing log data.

  • Fixed saving of pyro configuration that ended with 'Descending'. This would cause the configuration to revert to the previous state during setup.

The latest AltosUI and TeleGPS applications have improved functionality for analyzing flight data. The built-in graphing capabilities are improved with:

  • Graph lines have improved appearance to make them easier to distinguish. Markers may be placed at data points to show captured recorded data values.

  • Graphing offers the ability to adjust the smoothing of computed speed and acceleration data.

Exporting data for other applications has some improvements as well:

  • KML export now reports both barometric and GPS altitude data to make it more useful for Tripoli record reporting.

  • CSV export now includes TeleMega/EasyMega pyro voltages and tilt angle.

Categories: FLOSS Project Planets

Drupal blog: Accelerate Drupal 8 by funding a Core Committer

Planet Drupal - Tue, 2017-12-12 11:44

This blog has been re-posted and edited with permission from Dries Buytaert's blog. Please leave your comments on the original post.

We have ambitious goals for Drupal 8, including new core features such as Workspaces (content staging) and Layout Builder (drag-and-drop blocks), completing efforts such as the Migration path and Media in core, automated upgrades, and adoption of a JavaScript framework.

I met with several of the coordinators behind these initiatives. Across the board, they identified the need for faster feedback from Core Committers, citing that a lack of Committer time was often a barrier to the initiative's progress.

We have worked hard to scale the Core Committer Team. When Drupal 8 began, it was just catch and myself. Over time, we added additional Core Committers, and the team is now up to 13 members. We also added the concept of Maintainer roles to create more specialization and focus, which has increased our velocity as well.

I recently challenged the Core Committer Team and asked them what it would take to double their efficiency (and improve the velocity of all other core contributors and core initiatives). The answer was often straightforward; more time in the day to focus on reviewing and committing patches.

Most don't have funding for their work as Core Committers. It's something they take on part-time or as volunteers, and it often involves having to make trade-offs regarding paying work or family.

Of the 13 members of the Core Committer Team, three people noted that funding could make a big difference in their ability to contribute to Drupal 8, and could therefore help them empower others:

  • Lauri 'lauriii' Eskola, Front-end Framework Manager — Lauri is deeply involved with both the Out-of-the-Box Experience and the JavaScript Framework initiatives. In his role as front-end framework manager, he also reviews and unblocks patches that touch CSS/JS/HTML, which is key to many of the user-facing features in Drupal 8.5's roadmap.
  • Francesco 'plach' Placella, Framework Manager — Francesco has extensive experience in the Entity API and multilingual initiatives, making him an ideal reviewer for initiatives that touch lots of moving parts such as API-First and Workflow. Francesco was also a regular go-to for the Drupal 8 Accelerate program due to his ability to dig in on almost any problem.
  • Roy 'yoroy' Scholten, Product Manager — Roy has been involved in UX and Design for Drupal since the Drupal 5 days. Roy's insights into usability best practices and support and mentoring for developers is invaluable on the core team. He would love to spend more time doing those things, ideally supported by a multitude of companies each contributing a little, rather than just one.

Funding a Core Committer is one of the most high-impact ways you can contribute to Drupal. If you're interested in funding one or more of these amazing contributors, please contact me and I'll get you in touch with them.

Note that there is also ongoing discussion in Drupal.org's issue queue about how to expose funding opportunities for all contributors on Drupal.org.

Categories: FLOSS Project Planets

Django Weblog: DSF travel grants available for PyCon Namibia 2018

Planet Python - Tue, 2017-12-12 11:37
About PyCon Namibia

PyCon Namibia held its first edition in 2015.

The conference has been held annually since then, and has been at the heart of a new open-source software movement in Namibia. In particular, through PyNam, the Namibian Python Society, Python has become the focus of self-organised community volunteering activity in schools and universities.

In the last two years, assisted greatly by Helen Sherwood-Taylor, Django Girls has become an important part of the event too.

PyCons in Africa

The conference has also been the direct prompt for further new PyCons across Africa; Zimbabwe in 2016, Nigeria in 2017 and a planned PyCon Ghana next year. In each case, PyCon attendees from another country have returned home to set up their own events.

An important aspect of these events is the opportunity to establish relationships with the international community. Numerous people have travelled from other corners of the world to meet African programmers in their own countries, and many have returned multiple times.

Be a Pythonista, not a tourist

There is enormous value in this exchange, which gives Python/Django programmers from beyond Africa a unique opportunity to encounter African programmers in their own country, and to visit not as passing tourists but as Pythonistas and Djangonauts who will form long-term relationships with their African counterparts. This helps ensure that the international Python community meaningfully includes its members, wherever in the world they may be, and represents a chance like no other to understand them and what Python might mean in Africa.

There is probably no better way to understand what Python might mean in Namibia, for example, than having lunch with a group of Namibian high-school pupils and hearing about their ideas and plans for programming.

This exchange enriches not only the PyCon itself, but also the lives of the Pythonistas that it embraces, from both countries, and the communities they are a part of.

About the travel fund

In order to help maintain this valuable exchange between international Python communities, the Django Software Foundation has set aside a total of US$1500 to help enable travellers from abroad to visit Namibia for next year's PyCon, 20th-22nd February.

The DSF seeks expressions of interest from members of the international Django community who'd like to take advantage of these funds.

Please get in touch with us by email. We'd like to know:

  • who you are
  • why you'd like to participate
  • where you are travelling from and how much you estimate you will need

PyCon Namibia will benefit most from attendees who are interested in developing long-term relationships with its community and attendees.

See the conference website for information about travel and more.

Categories: FLOSS Project Planets

Kushal Das: Qubes OS 4.0rc3 and latest UEFI systems

Planet Python - Tue, 2017-12-12 10:19

Last week I received a new laptop, I am going to use it as my primary work station. The first step was to install Qubes OS 4.0rc3 on the system. It is a Thinkpad T470 with 32GB RAM and a SSD drive.

How to install Qubes on the latest UEFI systems?

A few weeks back, a patch was merged to the official Qubes documentation, which explains in clear steps how to create a bootable USB drive on a Fedora system using livecd-tools. Please follow the guide and create a USB drive which will work on these latest machines. Just simply using dd will not help.

First step after installing Qubes

I upgraded the dom0 to the current testing packages using the following command.

$ sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing $ sudo qubes-dom0-update qubes-template-fedora-26

I also installed the Fedora 26 template on my system using the next command. One of the important point to remember that Fedora 25 is going to be end of life today. So, better to use updated version of the distribution :)

There was another important thing happened in the last two weeks. I was in the Freedom of the Press Foundation office in San Fransisco. Means not only I managed to meet my amazing team, I also met many of my personal heroes in this trip. I may write a separate blog post about that later. But for now I can say that I managed to sit near to Micah Lee for 2 weeks and learn a ton about various things, including his Qubes workflow. The following two things were the first change I did to my installation (with his guidance) to make things working properly.

How to modify the copy-paste between domains shortcuts?

Generally Ctrl+Shift+c and Ctrl+Shift+v are used to copy-paste securely between different domains. But, those are the shortcuts to copy-paste from the terminal in all the systems. So, modifying them to a different key combination is very helpful for the muscle memory :)

Modify the following lines in the /etc/qubes/guid.conf file in dom0, I did a reboot after that to make sure that I am using this new key combination.

secure_copy_sequence = “Mod-c”; secure_paste_sequence = “Mod-v”;

The above configuration will modify the copy paste shortcuts to Windows+c and Windows+v in my keyboard layout.

Fixing the wireless driver issue in suspend/resume

I also found that if I suspend the system, after starting it on again, the wireless device was missing from the sys-net domain. Adding the following two module in the /rw/config/suspend-module-blacklist file on the sys-net domain helped me to fix that.

iwlmvm iwlwifi

The official documentation has a section on the same.

You can follow my posts on Qubes OS here.

Categories: FLOSS Project Planets

Continuum Analytics Blog: Anaconda Welcomes Lars Ewe as SVP of Engineering

Planet Python - Tue, 2017-12-12 09:00
Anaconda, Inc., provider of the most popular Python data science platform, today announced Lars Ewe as the company’s new senior vice president (SVP) of engineering. With more than 20 years of enterprise engineering experience, Ewe brings a strong foundation in big data, real-time analytics and security. He will lead the Anaconda Enterprise and Anaconda Distribution engineering teams.
Categories: FLOSS Project Planets

Droptica: Droptica: CKEditor in Drupal 8 Configuring and adding new options pt. I 

Planet Drupal - Tue, 2017-12-12 07:33
What is the CKEditor? It is one of many visual HTML editors. It allows you to easily enter text using an interface that resembles such editors as OpenOffice.  It makes text formatting easier through a set of buttons that serve, among others, for things such as: changing the font, text size, adding an image, making lists, and many others. The CKEditor is the default text editor in the Drupal 8 system. You do not need to add any libraries or modules in order to use it.  Enabled editor in the contend adding form By default, the editor is enabled for two text input formats:
Categories: FLOSS Project Planets
Syndicate content