FLOSS Project Planets

DataCamp: DataCamp and Springboard Are Working Together To Get You a Data Science Job!

Planet Python - Mon, 2017-09-18 09:18

DataCamp and Springboard are coming together to advance learning and career outcomes for aspiring data scientists.  

Joining forces was an obvious choice. Springboard’s human-centered approach to online learning perfectly complemented DataCamp’s expertise in interactive learning exercises. Together, we’ve created the Data Science Career Track, the first mentor-led data science bootcamp to come with a job guarantee. 

Each student in the Data Science Career Track will be assigned a personal industry mentor who’ll advise them on technical skills, project execution, and career advancement. Springboard’s expert-curated data science curriculum will be paired with DataCamp’s interactive exercises for a seamless learning experience. Finally, a career coach will work with students on interview skills, resume building, and personalized job searches to help them find the ideal data science position. 

The course is selective: about 18% of applicants are allowed to enroll after going through the admission process.

For eligible students, the course guarantees that you’ll find a job within six months after graduation or your money back.  

For a limited time only (until October 16th), you can use the code LOVEDATA to get $200 off if the Data Science Career Track. Click here for more information

Categories: FLOSS Project Planets

Doug Hellmann: gc — Garbage Collector — PyMOTW 3

Planet Python - Mon, 2017-09-18 09:00
gc exposes the underlying memory management mechanism of Python, the automatic garbage collector. The module includes functions for controlling how the collector operates and to examine the objects known to the system, either pending collection or stuck in reference cycles and unable to be freed. Read more… This post is part of the Python Module …
Categories: FLOSS Project Planets

Appnovation Technologies: Appnovator Spotlight: Richard Hales

Planet Drupal - Mon, 2017-09-18 08:41
Appnovator Spotlight: Richard Hales Who are you? What's your story?  Richard Hales - "Talent Acquisition". With 13 years in recruitment, working with some top global brands, hiring some of the best talent. I now use that experience and those connections to find the very best people for Appnovation What's your role at Appnovation?  Working in the UK, I am ideally placed to serv...
Categories: FLOSS Project Planets

Mike Driscoll: PyDev of the Week: Daniel Roseman

Planet Python - Mon, 2017-09-18 08:30

This week we welcome Daniel Roseman as our PyDev of the Week. I stumbled across Daniel on StackOverflow via some of the Python answers he has given. He is in the top 0.01% overall on StackOverflow, which is pretty impressive. He also has an old blog with a few interesting Python related articles in it. You can see what he’s been up to lately over on Github. Let’s take a few moments to get to know Daniel better!

Can you tell us a little about yourself (hobbies, education, etc):

I’m a self-taught programmer – my degree is actually in French – and I spent ten years working as a journalist and sub-editor before finally making the move into professional web development.

Since then I’ve worked at Global Radio, Glasses Direct, Google, and now the UK’s Government Digital Service, where I’m currently a technical architect on the publishing platform for the GOV.UK website.

Outside of work I’m a singer in various amateur choirs. I’ve also been running a Code Club at a local primary school for several years, helping ten and eleven year olds with their first introduction to programming using Scratch and later Python itself.

Why did you start using Python?

I got involved in helping out with a website for a charity, which was originally written in Python using Zope 2. Until then I’d never done any Python, and one of the original developers (thanks, Yoz!) helped me get started and pointed me towards Dive Into Python, which was an excellent resource for learning the language.

The charity site was quite basic at that time and didn’t have a proper CMS, so I looked around for technologies to make it more usable. That’s how I discovered Django, which was just then beginning to make an impact; this was around the time of the earliest open-source releases, version 0.90 or so. I fell in love with Django and was quickly able to use it to rebuild the site completely, and I’ve never looked back.

What other programming languages do you know and which is your favorite?

Most of my current team’s work is in Ruby, so professionally I’ve been mainly doing that for the last three years or so. There’s also some Go, although I haven’t done much there myself.

Python definitely remains my favourite. Although I do like a lot of things that Ruby brings, Python is still the language that fits my brain best.

What projects are you working on now?

I don’t get a lot of time for real open-source work because of family stuff and other commitments, so I tend to just contribute various bug fixes and minor features when I can.

One current project though is to see if I can use my experience with Code Club to write an introduction for kids to web development with Django. There are a few kids’ programming tutorials using Python, but nothing specifically focused on the web. It’s mainly inspired by the fantastic Django Girls tutorial, but I want to see if it’s possible to do an introduction from the ground up to all the relevant technologies for a much younger age group. It’s a long-term project though, so it’ll be a while before there’s anything ready to show.

Which Python libraries are your favorite (core or 3rd party)?

Obviously I’d put Django high up there on my list of favourites. It’s what got me properly into Python, and helped me find my first jobs in web development. There’s a great mix of usability and functionality, as well as a huge amount of third-party packages for just about anything.

How did you end up becoming one of the top “gurus” on StackOverflow for Python?

Persistence, and more than a little of “Someone is wrong on the Internet” syndrome. Like many programmers I do like to help and share my knowledge, and contributing to SO has been a really good way for me to do that: hopefully I’ve helped many many people there. And I get a lot of satisfaction from helping people who are trying their best to make something work, but have somehow misunderstood a concept or struggle to see why things aren’t doing what they think they should.

On top of that, I do like to write, but I rarely get the opportunity to sit down and write long articles or blog posts; but answering a question on SO with an explanation or code snippet takes only a minute or so. In effect, helping people on StackOverflow is my main contribution to the open source community.

For those wondering how I manage to answer so many questions, the feed of most recent Python and Django questions is in my RSS reader; so I often encounter a question I’d like to answer while I’m just browsing on the train to work, for example. I’ve become quite good at entering code examples using my phone keyboard.

What do you like the most about StackOverflow versus other tech help websites?

Mainly the direct focus on actual programming questions and answers. There’s a very clear idea of what is on- and off-topic there, and anything that isn’t an actual question about how to solve a specific programming problem quickly gets closed. Similarly, it enables and encourages posters to go back and edit their questions to post relevant details they may have missed out, making them more relevant and clearer.

Of course, the flip side of this is that it does cause it does sometimes appear unwelcoming to newcomers, who often don’t know exactly how to ask questions and get defensive when asked for more details. I’ve given a short talk at a couple of meetups about what exactly does make a good question and how to maximise the possibility you’ll get an answer; the slides are here: https://www.slideshare.net/danielroseman/asking-good-questions-53621064

On the other hand, there are a few things I don’t like. One of them, perhaps surprisingly, is the points system; I have far too many points. While that is to a certain extent because I contribute a lot, it’s also not insignificantly due to the fact that I joined early and wrote some “canonical” answers that get voted up a lot, even years later. Some of those answers aren’t even very good, but they continue to get votes precisely because they already have votes. I’m not really sure how this could be improved, though.

Thanks for doing the interview!

Categories: FLOSS Project Planets

Django Weekly: DjangoWeekly 56 - Free continuous delivery eBook from GoCD, A Complete Beginner's Guide to Django 2

Planet Python - Mon, 2017-09-18 08:04
Worthy Read
Free continuous delivery eBook from GoCDThis free reference guide will take you back to the basics. You’ll find visuals and definitions on key concepts and questions you need to answer about your teams to determine your readiness for continuous delivery. Download and share with your team.
advert, GoCD
A Complete Beginner's Guide to Django - Part 2Welcome to the second part of our Django Tutorial! In the previous lesson, we installed everything that we needed. Hopefully, you are all setup with Python 3.6 installed and Django 1.11 running inside a Virtual Environment. We already created the project we are going to play around. In this lesson, we are going to keep writing code in the same project.
tutorial
Load Testing a Django Application using LocustIOLocustIO, an open source tool written in python, is used for load testing of web applications. It is simple and easy to use with web UI to view the test results. It is scalable and can be distributed over multiple machines. This article demonstrates an example to use locust for load testing of our django web application.
django
Django Girls Impact Report 2016-2017This Impact Report aims to celebrate achievements of the Django Girls community in the past two years, and showcase the incredible growth of the organization. For the first time ever, we're also presenting results of a survey we conducted with almost 600 past Django Girls attendees to see if Django Girls Foundation actually achieves the goal of our mission: to bring more women into tech industry!
django-girls
Simple Nested API Using Django REST FrameworkIn this article you will learn how to build a simple REST API using Django REST Framework. The code in this article was written with Python 3.6, Django 1.11 and DRF 3.6 in mind.
DRF
Token Authentication and Authorization with GraphQL and DjangoIn my case, I wanted to use my existing Django Rest Framework (DRF) Token authentication endpoints alongside GraphQL. I'll be using a class-based view approach for Django, DRF, and Graphene.
GraphQL, token auth
Retrying Asynchronous Tasks With CeleryWriting resilient code that can handle task failure is important for maintaining modern functional systems. We’ll be going over how to retry asynchronous tasks with celery in python, commonly used in django applications.
celery
The India edition of Two Scoops of Django 1.11 is available on Flipkart and Amazon!The Indian Edition of the awesome Two Scoops of Django 1.11 is now on Flipkart and Amazon. Rejoice Django Developers from Indian.
book
Compare yourself to over 1,000 DevOps peers to see how they manage their processesHow do you compare?
advert
Embed docs directly on your website with a few lines of codeTest the API for free.
advert
How to Deploy Django Applications to AWS Using NanoboxIn this article, I'm going to walk through deploying a Django application to AWS using Nanobox. Nanobox uses Docker to provision local development environments, local staging environments, and scalable, highly-available production environments on AWS.
deployment
Multitenancy: juggling customer data in DjangoSuppose you want to build a new SaaS (Software as a Service) application. Suppose your application will store sensitive data from your customers. What is the best way to guarantee the isolation of the data and make sure information from one client does not leak to the other? The answer to that is: it depends. It depends on the number of customers you are planning to have. It depends on the size of your company. It depends on the technical skills of the developers working on the platform. And it depends on how sensitive the data is. In this article, I'll go over some of the architectures you can adopt in your application to tackle this problem and how to apply them in Django.
multitenancy
Obey the Testing Goat! Second Edition is outThe book is available both for free and for money. It's all about TDD and Web programming. Read it here!.
test driven development
Django Multiprocessingmultiprocessing
Use corresponding serializer class for different request method in Django Rest FrameworkDjango Rest Framework(DRF) provide a extremely convenience way to develop RESTful apps. Such as generics module, which contains many useful APIView based on the request method.
DRF
My Django Docker imageA description, step by step of how I builded my docker django image, how i loaded it on docker hub and how can be use it and customized.
docker, dockerfile

Projects
django-clever-cache - 0 Stars, 0 ForkDjango cache backend with automatic granular invalidation.
django-simple-affiliate - 0 Stars, 0 ForkThis is a very simple library that can be used to provide affiliate links in your django application. It is intentionally very lightweight, allowing your application to do whatever it wants with the data.
Categories: FLOSS Project Planets

Vardot: What Is A Chatbot, And Why Is It Important For Your Business?

Planet Drupal - Mon, 2017-09-18 07:06
What Is A Chatbot, And Why Is It Important For Your Business? Dmitrii Susloparov Mon, 09/18/2017 - 14:06

There is a great deal of enthusiasm surrounding chatbots in the Internet technology world today. Fanning the flames was the news that the White House had created a Facebook chatbot using Drupal. This post explains what a chatbot is, its current status, and how it can benefit business enterprises.

What is a chatbot?

Chatbots are software agents which communicate and collaborate with human users through text messaging using a natural language, say English, to accomplish specific tasks. Examples of common tasks in a business context are product inquiries, ordering, and troubleshooting.

 

Chatbots holds the promise of being the next generation of technology that people use to interact online with business enterprises. From a historical perspective, the first generation of customer contact technology involves websites. Users opens the company website within their browser, navigate web pages to get the information they want and to trigger various e-commerce transactions, such as ordering a product. Next up are mobile apps which users can download on their smartphones or tablets. The problem with apps is that people have to manually download and learn to use each of them. Chatbots lead the way for the next wave of technology. With chatbots, there are no new apps to download. This is because most users already have at least 1 instant messaging application installed on their communication devices, e.g., SMS, Facebook Messenger, Slack, Telegram, Kik, etc. Another advantage for chatbots is that, because chatbots communicate using a natural language, users don't need to learn yet another new website interface and to get comfortable with the unavoidable quirks.

 

The chatbot interface is powered by Artificial Intelligence (AI) technologies. AI is tasked to understand the text that users enter and pass on the knowledge to the backend for processing. Another benefit of using AI is that the chatbot will learn over time to better understand user preferences and as a result, deliver better and faster services.

Why are chatbots important to enterprises?

It was reported that, in 2016, for the first time in Internet history, there were more people using messaging apps than social media. It follows that chat has outpaced social media to become the de facto standard in how mobile users want to make contact. Mobile users are known to be an impatient bunch, ready to abandon any website en masse if they are made to wait for mere seconds after their initial request. Is your business staffed properly to handle this 24x7 onslaught of customer product queries, sales orders, and support requests?

 

Image Source: Business Insider

 

Chatbots can be programmed to monitor and respond to those chat sessions that fall within their domain expertise, such as troubleshooting, return merchandise authorization (RMA), sales inquiry, etc. For chatbots to do their job, enterprises first need to capture the aforementioned domain knowledge in a knowledge base. Once the knowledge becomes accessible, chatbots can staff the all-important corporate functions 24x7.

 

By deploying chatbots, a business can save money by easing the staff head count while guaranteeing good service response time. Besides its always-on feature, another major advantage of using chatbots is the consistency in how your business processes are applied: chatbots will execute the business logic consistently in all customer contacts.

 

Chatbots, advanced that it is, is not the panacea of all enterprise customer service problems, nor does it completely replace the entire human work force. Human agents are still required to solve the more complex problems that are beyond the ability of chatbots. While chatbots can resolve the most basic troubleshooting tasks, second-level support technicians are still required to tackle the complicated product issues.

State of the union

The chatbot technology is still an emerging technology. There are many components that have to work together to make chatbots work. As of today, the technology stack is not standardized, and a clear market leader has not yet manifested itself.

 

Just as there are many messaging apps, there are as many, if not more, chatbot building platforms, each designed to work with a subset of specific messaging apps. Some messaging app vendors, such as Facebook and Telegram, also provide their own official chatbot building platforms. Besides those, there are other third-party chatbot platforms that support multiple messaging apps. For instance, Chatfuel is a chatbot platform that supports both Facebook and Telegram. The Microsoft Bot Framework supports Facebook Messenger, Slack, and SMS.

 

Most chatbot building platforms claim that chatbots can be developed in minutes with no coding required. While creating a chatbot may take only minutes, making it do something useful involves customization including configuring the AI front-end engine, the e-commerce and payment processor backend, etc. Given the myriad technical choices and possibilities, it is best to leverage professional help to guide the development of chatbots for your business enterprise.

Chatbots and Drupal

Many businesses have already crafted their online presence in the form of a website using an enterprise-class CMS technology, for instance, Drupal. The good news is that you can add chatbot technology to your existing technology infrastructure, rather than starting from scratch.

 

If you have already built an enterprise-class Drupal website, you are a one-step ahead of everyone else. The Drupal infrastructure is essentially a portal that captures your business logic, including the backend portion that interfaces with your e-commerce and other back office systems. Adding chatbots to your overall technology stack involves adding the proper middleware to connect your chatbot frontend with your Drupal-based business logic backend.

 

As stated in the previous section, chatbots itself is an emerging technology that may be outside the scope for most in-house development expertise. Adding the middleware to join together chatbots and your Drupal backend is an extra level of software complexity. This middleware framework is available from the Drupal community, but is currently in a very early stage for commercialization. To ensure success for your chatbot project, professional consulting is highly recommended.

 

If you require professional services, whether to build from scratch an enterprise-class Drupal website with chatbot integration, or to add chatbot capabilities to your existing Drupal platform, Vardot is pleased to offer such services from its Jordanian headquarters or its American and Egyptian regional offices. Contact us now for more details regarding your project!

Categories: FLOSS Project Planets

Deeson: Component driven front-end development

Planet Drupal - Mon, 2017-09-18 06:58

At Deeson we’ve been working on ways to develop our front-end independently from any back-end application.

Our front-end team has developed web applications using decoupled JavaScript frameworks including Angular and React but we’ve found that for many website projects a full web application is overkill and the traditional HTML templating approach is still the most efficient.

Our back-end application is usually Drupal but we’re increasingly using other frameworks such as Laravel or Symfony and would like to be able to use a consistent approach for our front-end teams. 

We’ve developed a system for this that allows modern build tools, practices Component Driven Development, allows the generation of living style guides and is agnostic to the back-end.

Component Driven Development 

A component, for us, is a collection of HTML, CSS and JS that goes together to form some display element. Consider this against a more traditional approach where HTML, CSS and JS are stored in separate global files, for example, in a global style.css and app.js files.

By grouping code together into components, we separate our application by the domain language, rather than arbitrarily by file extension. This isolates components and makes them easier to develop and maintain.

Components get named based on the domain language of the project and client. Components are not defined for the designer by the limitations and modeling of the application. This provides a common language for the designers, developers and client and reduces the chances of misunderstanding when it comes to the development of functionality.

Using the BEM approach to structuring CSS we isolate much of our CSS to specific components rather than continuously generalising CSS in the way a CSS framework like Bootstrap does. This isolates much of the CSS to a specific component giving us high cohesion and low coupling allowing for confident maintenance, removing much of the fear of wondering what effect changing a piece of CSS is will have on the whole site.

This better matches the way that we work where the complexity of our challenging designs mean rapid delivery using a CSS framework isn’t possible.

Living style guides

The output of our front-end development will include a style guide which will render each of our components into static pages.

Style guides are useful as they demonstrate the components independently of the specific implementation. This allows more rapid front-end development as front-end developers can work without having to worry about how the back-end will integrate.

Over time, however, these style guides move out of sync with the applications they were developed to provide styling information for. An application developer's job is to integrate the HTML provided by the style guide into the finished site. This has meant copying and pasting code out of the style guide templates and into the application’s templating system.

At this point we have duplication of code and the ability to maintain a strict style guide is lost. When the style guide is updated, all the application templates affected must be identified and updated as well.

Our approach makes the style guide a living style guide. The front-end templates we produce for our components get referenced directly from the target applications theme system. This means that our front-end templates are the exact same ones that the application will be using within the theme.

Front-end developers can make changes to it knowing that those changes will flow through into the application without need for a second step.

For Drupal developers this means either providing new theme functions for the front-end templates or referencing our front-end templates from within Drupal templates.

Modern build tools, agnostic to the back-end

Freed from the constraints of a specific application’s templating system we can select the most appropriate tools for the job of front-end development.

Our front-end tooling uses the latest standards and tools. We’re using yarn for package management, and webpack to bundle the static assets. 

Very little knowledge of the back-end is assumed or needed in this approach. You can confidently bring new front-end developers onto your team who are used to using the latest tools without having to spend the first few weeks teaching them the specific theming language and quirks of your back-end application such as the Drupal theme layer.

A real example 

We’ve got an exemplar project to showcase this way of working. If you clone the project at https://github.com/teamdeeson/cdd-demo and follow the instructions in the README you’ll get a Drupal 8 project up and running with a theme that uses this process. You’ll see we’ve developed a componentised version of the Bartik theme for this.

If you are intrigued by this and would like to hear more, you’ll enjoy my talk on the subject at DrupalCon Vienna on 26th September.

In summary

Component driven front-end development has worked well for us at Deeson, allowing rapid independent development of our front-end code. Our designers are freed from the previous constraints of designing for a Drupal website and our developers get to use the latest tools and get onboarded quicker.

Like the sound of the way we do things? We're currently hiring a Senior Front-end Developer.

Categories: FLOSS Project Planets

Reuven Lerner: My favorite terrible Python error message

Planet Python - Mon, 2017-09-18 03:39

Students in my Python classes occasionally get the following error message:

TypeError: object() takes no parameters

This error message is technically true, as I’ll explain in a moment. But it’s surprising and confusing for people who are new to Python, because it doesn’t point to the source of the actual problem.

Here’s the basic idea: Python methods are attributes, which means that when we invoke methods, Python needs to search for the attribute we’ve named. In other words, if I invoke:

o.m()

then Python will first look for the “m” attribute on the “o” object. If “o” has an attribute named “m” (i.e., if hasattr(o, ‘m’) returns True) then it retrieves the attribute’s value, and tries to call it.

However, Python methods aren’t defined on individual objects. They’re defined on classes. Which means that in almost all cases, if “m” is an actual method that can be invoked on “o”, there won’t be any “m” attribute on “o”.  Instead, we’ll need to look at type(o), the class to which “o” belongs, and look there.

And indeed, that’s how attributes work in Python: First search on the named object. If the attribute isn’t there, then look at the object’s class.  So we look for “m” on o’s class.  If the attribute is there, then it is invoked.  That’s what happens in normal method calls.

But say that the attribute isn’t on the class, either. What then? Python continues its search, looking next at the class from which type(o) inherits — which is located on the attribute type(o).__bases__.  This is a tuple, because Python classes can inherit from more than one parent; let’s ignore that for now.

Most classes inherit from the base object in the Python universe, known as “object”.  In Python 3, if you don’t specify “object” as the base from which you inherit, then it’s done for you automatically. In Python 2, failing to specify that a class inherits from “object” means that you have an “old-style class,” which will operate differently. I continue to specify “object” in my Python 3 classes, partly out of habit, partly because I think it looks nicer, and partly because I want my code to be compatible across versions as much as possible.

What happens if the attribute doesn’t exist on “object”?  Then we get an “attribute error,” with Python telling us that the attribute doesn’t exist.

However, this isn’t what happens in the case of the error message I showed:

TypeError: object() takes no parameters

This error message happens when you try to create a new instance of a class. For example:

class Foo(object):     pass

If I say

f = Foo()

then I don’t get any error message. But if I say

f = Foo(10)

then I get the TypeError.  Why?

Because Python objects are created in two stages: First, the object is created in the __new__ method. This is a method that we almost never want to write; let Python take care of the allocation and creation of new objects.

However, __new__ doesn’t immediately return the object that it has created. Rather, it first searches for an __init__ method, whose job is to add new attributes to the newly created object. How does it look for (and then invoke) __init__?  It turns to the new object, which I’ll call “o” here, and invokes

o.__init__()

So, what happens now? Python looks for “__init__” on “o”, but doesn’t find it.  It looks for “__init__” on type(o), aka the “Foo” class, and doesn’t find it.  So it keeps searching, and looks on “object” for an “__init__” attribute.

Good news: object.__init__ exists!  Moreover, it’s a method!  So Python tries to invoke it, passing the argument that I handed to Foo (i.e., 10).  But object.__init__ doesn’t take any arguments. And thus we get the error message

TypeError: object() takes no parameters

What’s especially confusing, for me and many of my students, is that Python doesn’t say, “object.__init__()” takes no parameters. So they’re not sure how object figures into this, or where their mistake might be.

After reading this, though, I’m hoping that you can guess what it means: Simply put, this error message says, “You forgot to define an __init__ method on your object.”  This can be out of forgetfulness, but I’ve also seen people forget one or more of the underscores on either side of “__init__”, or even (my favorite) define a method called “__int__”, which is great for converting objects into integers, but not for initializing attributes.

So, is the error message wrong? No, it’s perfectly logical. But as with many “perfectly logical” things, it makes sense after you are steeped in the overall logic of the system, and tends to confuse those who most need the help.

The post My favorite terrible Python error message appeared first on Lerner Consulting Blog.

Categories: FLOSS Project Planets

Catalin George Festila: YARA python module - part 002 .

Planet Python - Mon, 2017-09-18 01:43
This is another part of YARA python tutorial and the goal of this part is install the yara modules.
The YARA modules provides extending features to allow us to define data structures and functions which can be used in your rules to express more complex conditions.
You can also write your own modules.
Some known modules used by YARA are:
  • PE
  • ELF
  • Cuckoo
  • Magic
  • Hash
  • Math
First you need to install or reinstall YARA to the last version:
>>> yara.__version__
'3.6.3'The Cuckoo module enables you to create YARA rules based on behavioral information generated by a Cuckoo sandbox.
C:\Python27\Scripts>pip install yara-python
Collecting yara-python
Downloading yara_python-3.6.3-cp27-cp27m-win32.whl (606kB)
100% |################################| 614kB 1.3MB/s
Installing collected packages: yara-python
Successfully installed yara-python-3.6.3
pip install cuckoo
Collecting cuckoo
Downloading Cuckoo-2.0.4.4.tar.gz (3.1MB)
100% |################################| 3.1MB 255kB/s
...
Successfully installed Mako-1.0.7 alembic-0.8.8 androguard-3.0.1 beautifulsoup4-4.5.3 capstone-windows-3.0.4 chardet-2.3.0 click-6.6 colorama-0.3.7 cuckoo-2.0.4.4 django-1.8.4 django-extensions-1.6.7 dpkt-1.8.7 ecdsa-0.13 egghatch-0.2.1 elasticsearch-5.3.0 flask-sqlalchemy-2.1 httpreplay-0.2.1 jsbeautifier-1.6.2 jsonschema-2.6.0 olefile-0.43 oletools-0.42 peepdf-0.3.6 pefile2-1.2.11 pillow-3.2.0 pyelftools-0.24 pymisp-2.4.54 pymongo-3.0.3 python-dateutil-2.4.2 python-editor-1.0.3 python-magic-0.4.12 pythonaes-1.0 requests-2.13.0 sflock-0.2.16 sqlalchemy-1.0.8 tlslite-ng-0.6.0 unicorn-1.0.1 wakeonlan-0.2.2Let's test this python module:>>> import cuckoo
>>> from cuckoo import *
>>> dir(cuckoo)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', 'auxiliary', 'common', 'compat', 'core', 'machinery', 'misc', 'plugins', 'processing', 'reporting', 'signatures', 'web']Let's test some yara modules:
>>> import yara
>>> rule = yara.compile(source='import \"pe\"')
>>> rule = yara.compile(source='import \"elf\"')
>>> rule = yara.compile(source='import \"cuckoo\"')
>>> rule = yara.compile(source='import \"math\"')
I could not use the YARA modules: hash and magic.
I will solve this problem in the future.
You can also write your own modules ( see this webpage ).
Categories: FLOSS Project Planets

Bryan Pendleton: News of the weird, part 4 (of four)

Planet Apache - Sun, 2017-09-17 23:13

Well, this isn't exactly news, and I guess you'll have to judge for yourself whether it's weird or not.

But I thought both of these were pretty interesting.

  • How Half Of America Lost Its F**king MindThere's this universal shorthand that epic adventure movies use to tell the good guys from the bad. The good guys are simple folk from the countryside ...

    ... while the bad guys are decadent assholes who live in the city and wear stupid clothes.

    The theme expresses itself in several ways -- primitive vs. advanced, tough vs. delicate, masculine vs. feminine, poor vs. rich, pure vs. decadent, traditional vs. weird. All of it is code for rural vs. urban. That tense divide between the two doesn't exist because of these movies, obviously. These movies used it as shorthand because the divide already existed.

  • I Spent 5 Years With Some of Trump’s Biggest Fans. Here’s What They Won’t Tell You.Pervasive among the people I talked to was a sense of detachment from a distant elite with whom they had ever less contact and less in common.

    ...

    Trump has put on his blue-collar cap, pumped his fist in the air, and left mainstream Republicans helpless. Not only does he speak to the white working class’ grievances; as they see it, he has finally stopped their story from being politically suppressed. We may never know if Trump has done this intentionally or instinctively, but in any case he’s created a movement much like the anti-immigrant but pro-welfare-state right-wing populism on the rise in Europe. For these are all based on variations of the same Deep Story of personal protectionism.

Categories: FLOSS Project Planets

Russ Allbery: Consolidation haul

Planet Debian - Sun, 2017-09-17 20:34

My parents are less fond than I am of filling every available wall in their house with bookshelves and did a pruning of their books. A lot of them duplicated other things that I had, or didn't sound interesting, but I still ended up with two boxes of books (and now have to decide which of my books to prune, since I'm out of shelf space).

Also included is the regular accumulation of new ebook purchases.

Mitch Albom — Tuesdays with Morrie (nonfiction)
Ilona Andrews — Clean Sweep (sff)
Catherine Asaro — Charmed Sphere (sff)
Isaac Asimov — The Caves of Steel (sff)
Isaac Asimov — The Naked Sun (sff)
Marie Brennan — Dice Tales (nonfiction)
Captain Eric "Winkle" Brown — Wings on My Sleeve (nonfiction)
Brian Christian & Tom Griffiths — Algorithms to Live By (nonfiction)
Tom Clancy — The Cardinal of the Kremlin (thriller)
Tom Clancy — The Hunt for the Red October (thriller)
Tom Clancy — Red Storm Rising (thriller)
April Daniels — Sovereign (sff)
Tom Flynn — Galactic Rapture (sff)
Neil Gaiman — American Gods (sff)
Gary J. Hudson — They Had to Go Out (nonfiction)
Catherine Ryan Hyde — Pay It Forward (mainstream)
John Irving — A Prayer for Owen Meany (mainstream)
John Irving — The Cider House Rules (mainstream)
John Irving — The Hotel New Hampshire (mainstream)
Lawrence M. Krauss — Beyond Star Trek (nonfiction)
Lawrence M. Krauss — The Physics of Star Trek (nonfiction)
Ursula K. Le Guin — Four Ways to Forgiveness (sff collection)
Ursula K. Le Guin — Words Are My Matter (nonfiction)
Richard Matheson — Somewhere in Time (sff)
Larry Niven — Limits (sff collection)
Larry Niven — The Long ARM of Gil Hamilton (sff collection)
Larry Niven — The Magic Goes Away (sff)
Larry Niven — Protector (sff)
Larry Niven — World of Ptavvs (sff)
Larry Niven & Jerry Pournelle — The Gripping Hand (sff)
Larry Niven & Jerry Pournelle — Inferno (sff)
Larry Niven & Jerry Pournelle — The Mote in God's Eye (sff)
Flann O'Brien — The Best of Myles (nonfiction)
Jerry Pournelle — Exiles to Glory (sff)
Jerry Pournelle — The Mercenary (sff)
Jerry Pournelle — Prince of Mercenaries (sff)
Jerry Pournelle — West of Honor (sff)
Jerry Pournelle (ed.) — Codominium: Revolt on War World (sff anthology)
Jerry Pournelle & S.M. Stirling — Go Tell the Spartans (sff)
J.D. Salinger — The Catcher in the Rye (mainstream)
Jessica Amanda Salmonson — The Swordswoman (sff)
Stanley Schmidt — Aliens and Alien Societies (nonfiction)
Cecilia Tan (ed.) — Sextopia (sff anthology)
Lavie Tidhar — Central Station (sff)
Catherynne Valente — Chicks Dig Gaming (nonfiction)
J.E. Zimmerman — Dictionary of Classical Mythology (nonfiction)

This is an interesting tour of a lot of stuff I read as a teenager (Asimov, Niven, Clancy, and Pournelle, mostly in combination with Niven but sometimes his solo work).

I suspect I will no longer consider many of these books to be very good, and some of them will probably go back into used bookstores after I've re-read them for memory's sake, or when I run low on space again. But all those mass market SF novels were a big part of my teenage years, and a few (like Mote In God's Eye) I definitely want to read again.

Also included is a random collection of stuff my parents picked up over the years. I don't know what to expect from a lot of it, which makes it fun to anticipate. Fall vacation is coming up, and with it a large amount of uninterrupted reading time.

Categories: FLOSS Project Planets

Matthew Rocklin: Dask on HPC - Initial Work

Planet Python - Sun, 2017-09-17 20:00

This work is supported by Anaconda Inc. and the NSF EarthCube program.

We recently announced a collaboration between the National Center for Atmospheric Research (NCAR), Columbia University, and Anaconda Inc to accelerate the analysis of atmospheric and oceanographic data on high performance computers (HPC) with XArray and Dask. The full text of the proposed work is available here. We are very grateful to the NSF EarthCube program for funding this work, which feels particularly relevant today in the wake (and continued threat) of the major storms Harvey, Irma, and Jose.

This is a collaboration of academic scientists (Columbia), infrastructure stewards (NCAR), and software developers (Anaconda and Columbia and NCAR) to scale current workflows with XArray and Jupyter onto big-iron HPC systems and peta-scale datasets. In the first week after the grant closed a few of us focused on the quickest path to get science groups up and running with XArray, Dask, and Jupyter on these HPC systems. This blogpost details what we achieved and some of the new challenges that we’ve found in that first week. We hope to follow this blogpost with many more to come in the future. Today we cover the following topics:

  1. Deploying Dask with MPI
  2. Interactive deployments on a batch job scheduler, in this case PBS
  3. The virtues of JupyterLab in a remote system
  4. Network performance and 3GB/s infiniband
  5. Modernizing XArray’s interactions with Dask’s distributed scheduler

A video walkthrough deploying Dask on XArray on an HPC system is available on YouTube and instructions for atmospheric scientists with access to the Cheyenne Supercomputer is available here.

Now lets start with technical issues:

Deploying Dask with MPI

HPC systems use job schedulers like SGE, SLURM, PBS, LSF, and others. Dask has been deployed on all of these systems before either by academic groups or financial companies. However every time we do this it’s a little different and generally tailored to a particular cluster.

We wanted to make something more general. This started out as a GitHub issue on PBS scripts that tried to make a simple common template that people could copy-and-modify. Unfortunately, there were significant challenges with this. HPC systems and their job schedulers seem to focus and easily support only two common use cases:

  1. Embarrassingly parallel “run this script 1000 times” jobs. This is too simple for what we have to do.
  2. MPI jobs. This seemed like overkill, but is the approach that we ended up taking.

Deploying dask is somewhere between these two. It falls into the master-slave pattern (or perhaps more appropriately coordinator-workers). We ended up building an MPI4Py program that launches Dask. MPI is well supported, and more importantly consistently supported, by all HPC job schedulers so depending on MPI provides a level of stability across machines. Now dask.distributed ships with a new dask-mpi executable:

mpirun --np 4 dask-mpi

To be clear, Dask isn’t using MPI for inter-process communication. It’s still using TCP. We’re just using MPI to launch a scheduler and several workers and hook them all together. In pseudocode the dask-mpi executable looks something like this:

from mpi4py import MPI comm = MPI.COMM_WORLD rank = comm.Get_rank() if rank == 0: start_dask_scheduler() else: start_dask_worker()

Socially this is useful because every cluster management team knows how to support MPI, so anyone with access to such a cluster has someone they can ask for help. We’ve successfully translated the question “How do I start Dask?” to the question “How do I run this MPI program?” which is a question that the technical staff at supercomputer facilities are generally much better equipped to handle.

Working Interactively on a Batch Scheduler

Our collaboration is focused on interactive analysis of big datasets. This means that people expect to open up Jupyter notebooks, connect to clusters of many machines, and compute on those machines while they sit at their computer.

Unfortunately most job schedulers were designed for batch scheduling. They will try to run your job quickly, but don’t mind waiting for a few hours for a nice set of machines on the super computer to open up. As you ask for more time and more machines, waiting times can increase drastically. For most MPI jobs this is fine because people aren’t expecting to get a result right away and they’re certainly not interacting with the program, but in our case we really do want some results right away, even if they’re only part of what we asked for.

Handling this problem long term will require both technical work and policy decisions. In the short term we take advantage of two facts:

  1. Many small jobs can start more quickly than a few large ones. These take advantage of holes in the schedule that are too small to be used by larger jobs.
  2. Dask doesn’t need to be started all at once. Workers can come and go.

And so I find that if I ask for several single machine jobs I can easily cobble together a sizable cluster that starts very quickly. In practice this looks like the following:

$ qsub start-dask.sh # only ask for one machine $ qsub add-one-worker.sh # ask for one more machine $ qsub add-one-worker.sh # ask for one more machine $ qsub add-one-worker.sh # ask for one more machine $ qsub add-one-worker.sh # ask for one more machine $ qsub add-one-worker.sh # ask for one more machine $ qsub add-one-worker.sh # ask for one more machine

Our main job has a wall time of about an hour. The workers have shorter wall times. They can come and go as needed throughout the computation as our computational needs change.

Jupyter Lab and Web Frontends

Our scientific collaborators enjoy building Jupyter notebooks of their work. This allows them to manage their code, scientific thoughts, and visual outputs all at once and for them serves as an artifact that they can share with their scientific teams and collaborators. To help them with this we start a Jupyter server on the same machine in their allocation that is running the Dask scheduler. We then provide them with SSH-tunneling lines that they can copy-and-paste to get access to the Jupyter server from their personal computer.

We’ve been using the new Jupyter Lab rather than the classic notebook. This is especially convenient for us because it provides much of the interactive experience that they lost by not working on their local machine. They get a file browser, terminals, easy visualization of textfiles and so on without having to repeatedly SSH into the HPC system. We get all of this functionality on a single connection and with an intuitive Jupyter interface.

For now we give them a script to set all of this up. It starts Jupyter Lab using Dask and then prints out the SSH-tunneling line.

from dask.distributed import Client client = Client(scheduler_file='scheduler.json') import socket host = client.run_on_scheduler(socket.gethostname) def start_jlab(dask_scheduler): import subprocess proc = subprocess.Popen(['jupyter', 'lab', '--ip', host, '--no-browser']) dask_scheduler.jlab_proc = proc client.run_on_scheduler(start_jlab) print("ssh -N -L 8787:%s:8787 -L 8888:%s:8888 -L 8789:%s:8789 cheyenne.ucar.edu" % (host, host, host))

Long term we would like to switch to an entirely point-and-click interface (perhaps something like JupyterHub) but this will requires additional thinking about deploying distributed resources along with the Jupyter server instance.

Network Performance on Infiniband

The intended computations move several terabytes across the cluster. On this cluster Dask gets about 1GB/s simultaneous read/write network bandwidth per machine using the high-speed Infiniband network. For any commodity or cloud-based system this is very fast (about 10x faster than what I observe on Amazon). However for a super-computer this is only about 30% of what’s possible (see hardware specs).

I suspect that this is due to byte-handling in Tornado, the networking library that Dask uses under the hood. The following image shows the diagnostic dashboard for one worker after a communication-heavy workload. We see 1GB/s for both read and write. We also see 100% CPU usage.

Network performance is a big question for HPC users looking at Dask. If we can get near MPI bandwidth then that may help to reduce concerns for this performance-oriented community.

How do I use Infiniband network with Dask?

XArray and Dask.distributed

XArray was the first major project to use Dask internally. This early integration was critical to prove out Dask’s internals with user feedback. However it also means that some parts of XArray were designed well before some of the newer parts of Dask, notably the asynchronous distributed scheduling features.

XArray can still use Dask on a distributed cluster, but only with the subset of features that are also available with the single machine scheduler. This means that persisting data in distributed RAM, parallel debugging, publishing shared datasets, and so on all require significantly more work today with XArray than they should.

To address this we plan to update XArray to follow a newly proposed Dask interface. This is complex enough to handle all Dask scheduling features, but light weight enough not to actually require any dependence on the Dask library itself. (Work by Jim Crist.)

We will also eventually need to look at reducing overhead for inspecting several NetCDF files, but we haven’t yet run into this, so I plan to wait.

Future Work

We think we’re at a decent point for scientific users to start playing with the system. We have a Getting Started with Dask on Cheyenne wiki page that our first set of guinea pig users have successfully run through without much trouble. We’ve also identified a number of issues that the software developers can work on while the scientific teams spin up.

  1. Zero copy Tornado writes to improve network bandwidth
  2. Enable Dask.distributed features in XArray by formalizing dask’s expected interface
  3. Dynamic deployments on batch job schedulers

We would love to engage other collaborators throughout this process. If you or your group work on related problems we would love to hear from you. This grant isn’t just about serving the scientific needs of researchers at Columbia and NCAR, but about building long-term systems that can benefit the entire atmospheric and oceanographic community. Please engage on the Pangeo GitHub issue tracker.

Categories: FLOSS Project Planets

Sean Whitton: Debian Policy call for participation -- September 2017

Planet Debian - Sun, 2017-09-17 19:04

Here’s a summary of the bugs against the Debian Policy Manual. Please consider getting involved, whether or not you’re an existing contributor.

Consensus has been reached and help is needed to write a patch

#172436 BROWSER and sensible-browser standardization

#273093 document interactions of multiple clashing package diversions

#299007 Transitioning perms of /usr/local

#314808 Web applications should use /usr/share/package, not /usr/share/doc/…

#425523 Describe error unwind when unpacking a package fails

#452393 Clarify difference between required and important priorities

#476810 Please clarify 12.5, “Copyright information”

#484673 file permissions for files potentially including credential informa…

#491318 init scripts “should” support start/stop/restart/force-reload - why…

#556015 Clarify requirements for linked doc directories

#568313 Suggestion: forbid the use of dpkg-statoverride when uid and gid ar…

#578597 Recommend usage of dpkg-buildflags to initialize CFLAGS and al.

#582109 document triggers where appropriate

#587991 perl-policy: /etc/perl missing from Module Path

#592610 Clarify when Conflicts + Replaces et al are appropriate

#613046 please update example in 4.9.1 (debian/rules and DEB_BUILD_OPTIONS)

#614807 Please document autobuilder-imposed build-dependency alternative re…

#628515 recommending verbose build logs

#664257 document Architecture name definitions

#682347 mark ‘editor’ virtual package name as obsolete

#685506 copyright-format: new Files-Excluded field

#685746 debian-policy Consider clarifying the use of recommends

#688251 Built-Using description too aggressive

#749826 [multiarch] please document the use of Multi-Arch field in debian/c…

#757760 please document build profiles

#759316 Document the use of /etc/default for cron jobs

#761219 document versioned Provides

#767839 Linking documentation of arch:any package to arch:all

#770440 policy should mention systemd timers

#773557 Avoid unsafe RPATH/RUNPATH

#780725 PATH used for building is not specified

#793499 The Installed-Size algorithm is out-of-date

#810381 Update wording of 5.6.26 VCS-* fields to recommend encryption

#823256 Update maintscript arguments with dpkg >= 1.18.5

#833401 virtual packages: dbus-session-bus, dbus-default-session-bus

#835451 Building as root should be discouraged

#838777 Policy 11.8.4 for x-window-manager needs update for freedesktop menus

#845715 Please document that packages are not allowed to write outside thei…

#853779 Clarify requirements about update-rc.d and invoke-rc.d usage in mai…

#874019 Note that the ’-e’ argument to x-terminal-emulator works like ’–’

#874206 allow a trailing comma in package relationship fields

Wording proposed, awaiting review from anyone and/or seconds by DDs

#515856 remove get-orig-source

#542288 Versions for native packages, NMU’s, and binary only uploads

#582109 document triggers where appropriate

#610083 Remove requirement to document upstream source location in debian/c…

#645696 [copyright-format] clearer definitions and more consistent License:…

#649530 [copyright-format] clearer definitions and more consistent License:…

#662998 stripping static libraries

#682347 mark ‘editor’ virtual package name as obsolete

#683222 say explicitly that debian/changelog is required in source packages

#688251 Built-Using description too aggressive

#737796 copyright-format: support Files: paragraph with both abbreviated na…

#756835 Extension of the syntax of the Packages-List field.

#786470 [copyright-format] Add an optional “License-Grant” field

#810381 Update wording of 5.6.26 VCS-* fields to recommend encryption

#835451 Building as root should be discouraged

#845255 Include best practices for packaging database applications

#850729 Documenting special version number suffixes

#874090 Clarify wording of some passages

#874095 copyright-format: Use the “synopsis” term established in the de…

Merged for the next release

#661928 recipe for determining shlib package name

#679751 please clarify package account and home directory location in policy

#683222 say explicitly that debian/changelog is required in source packages

#870915 [5.6.30] Testsuite: There are much more defined values

#872893 Chapters, sections, appendices and numbering

#872895 Include multi-page HTML in package

#872896 An html.tar.gz has leaked into the .deb?

#872900 Very generic info file name

#872950 Too much indirection in info file menus

#873819 upgrading-checklist.txt: typo pgpsignurlmangle in section 4.11 of V…

#874411 missing line breaks in summary of ways maintainers scripts are call…

Categories: FLOSS Project Planets

Last week in Kube

Planet KDE - Sun, 2017-09-17 18:20

“Kube is a modern communication and collaboration client built with QtQuick on top of a high performance, low resource usage core. It provides online and offline access to all your mail, contacts, calendars, notes, todo’s and more. With a strong focus on usability, the team works with designers and UX experts from the ground up, to build a product that is not only visually appealing but also a joy to use.”

For more info, head over to: kube.kde.org

  • We now synchronize both contacts and drafts when opening the composer. The former are required for autocompletion, the latter to show your drafts.
  • A maillist filter bar to replace the currently defunct search.
  • We now synchronize the currently selected folder every 5min.
  • Bumped Qt requirement to 5.9 so we can finally use the focus stealing prevention of the webengineview.
  • Automatically launch into the account configuration if no accounts are set-up.
  • A single account mode for deployments that support only a single account. This can currently only be activated with a switch in the qml code.
  • Prototyped a simple html editor, unfortunately the QML TextArea API is just not there yet and the end result ended up buggy. Postponed for now.
  • Worked out a plan for gpg based encryption that is now available as phabricator tickets for the next milestone.
  • Improved input form validation and feedback.
  • Did some work towards improving a performance bottleneck in live queries when a lot (thousands) of updates are coming in (such as during the initial sync). Not quite there yet though.
  • Did some cross compilation experiments with MXE for Windows and without MXE for Android. WIP
  • Witnessed Kube on the Plasma Phone. This will require an adapted UI but generally seems to actually work.
  • Fixed some layouting issues where some TextAreas wouldn’t resize with the rest of the UI.

Kube Commits, Sink Commits

Previous updates


Categories: FLOSS Project Planets

Simple is Better Than Complex: A Complete Beginner's Guide to Django - Part 3

Planet Python - Sun, 2017-09-17 17:00
Introduction

In this tutorial, we are going to dive deep into two fundamental concepts: URLs and Forms. In the process, we are going to explore many other concepts like creating reusable templates and installing third-party libraries. We are also going to write plenty of unit tests.

If you are following this tutorial series since the first part, coding your project and following the tutorial step by step, you may need to update your models.py before starting:

boards/models.py

class Topic(models.Model): # other fields... # Add `auto_now_add=True` to the `last_updated` field last_updated = models.DateTimeField(auto_now_add=True) class Post(models.Model): # other fields... # Add `null=True` to the `updated_by` field updated_by = models.ForeignKey(User, null=True, related_name='+')

Now run the commands with the virtualenv activated:

python manage.py makemigrations python manage.py migrate

If you already have null=True in the updated_by field and the auto_now_add=True in the last_updated field, you can safely ignore the instructions above.

If you prefer to use my source code as a starting point, you can grab it on GitHub.

The current state of the project can be found under the release tag v0.2-lw. The link below will take you to the right place:

https://github.com/sibtc/django-beginners-guide/tree/v0.2-lw

The development will follow from here.

URLs

Proceeding with the development of our application, now we have to implement a new page to list all the topics that belong to a given Board. Just to recap, below you can see the wireframe we draw in the previous tutorial:

Figure 1: Boards project wireframe listing all topics in the Django board.

We will start by editing the urls.py inside the myproject folder:

myproject/urls.py

from django.conf.urls import url from django.contrib import admin from boards import views urlpatterns = [ url(r'^$', views.home, name='home'), url(r'^boards/(?P<pk>\d+)/$', views.board_topics, name='board_topics'), url(r'^admin/', admin.site.urls), ]

This time let’s take a moment and analyze the urlpatterns and url.

The URL dispatcher and URLconf (URL configuration) are fundamental parts of a Django application. In the beginning, it can look confusing; I remember having a hard time when I first started developing with Django.

In fact, right now the Django Developers are working on a proposal to make simplified routing syntax. But for now, as per the version 1.11, that’s what we have. So let’s try to understand how it works.

A project can have many urls.py distributed among the apps. But Django needs a url.py to use as a starting point. This special urls.py is called root URLconf. It’s defined in the settings.py file.

myproject/settings.py

ROOT_URLCONF = 'myproject.urls'

It already comes configured, so you don’t need to change anything here.

When Django receives a request, it starts searching for a match in the project’s URLconf. It starts with the first entry of the urlpatterns variable, and test the requested URL against each url entry.

If Django finds a match, it will pass the request to the view function, which is the second parameter of the url. The order in the urlpatterns matters, because Django will stop searching as soon as it finds a match. Now, if Django doesn’t find a match in the URLconf, it will raise a 404 exception, which is the error code for Page Not Found.

This is the anatomy of the url function:

def url(regex, view, kwargs=None, name=None): # ...
  • regex: A regular expression for matching URL patterns in strings. Note that these regular expressions do not search GET or POST parameter. In a request to http://127.0.0.1:8000/boards/?page=2 only /boards/ will be processed.
  • view: A view function used to process the user request for a matched URL. It also accepts the return of the django.conf.urls.include function, which is used to reference an external urls.py file. You can, for example, use it to define a set of app specific URLs, and include it in the root URLconf using a prefix. We will explore more on this concept later on.
  • kwargs: Arbitrary keyword arguments that’s passed to the target view. It is normally used to do some simple customization on reusable views. We don’t use it very often.
  • name: A unique identifier for a given URL. This is a very important feature. Always remember to name your URLs. With this, you can change a specific URL in the whole project by just changing the regex. So it’s important to never hard code URLs in the views or templates, and always refer to the URLs by its name.

Basic URLs

Basic URLs are very simple to create. It’s just a matter of matching strings. For example, let’s say we wanted to create an “about” page, it could be defined like this:

from django.conf.urls import url from boards import views urlpatterns = [ url(r'^$', views.home, name='home'), url(r'^about/$', views.about, name='about'), ]

We can also create deeper URL structures:

from django.conf.urls import url from boards import views urlpatterns = [ url(r'^$', views.home, name='home'), url(r'^about/$', views.about, name='about'), url(r'^about/company/$', views.about_company, name='about_company'), url(r'^about/author/$', views.about_author, name='about_author'), url(r'^about/author/vitor/$', views.about_vitor, name='about_vitor'), url(r'^about/author/erica/$', views.about_erica, name='about_erica'), url(r'^privacy/$', views.privacy_policy, name='privacy_policy'), ]

Those are some examples of simple URL routing. For all the examples above, the view function will follow this structure:

def about(request): # do something... return render(request, 'about.html') def about_company(request): # do something else... # return some data along with the view... return render(request, 'about_company.html', {'company_name': 'Simple Complex'}) Advanced URLs

A more advanced usage of URL routing is achieved by taking advantage of the regex to match certain types of data and create dynamic URLs.

For example, to create a profile page, like many services do like github.com/vitorfs or twitter.com/vitorfs, where “vitorfs” is my username, we can do the following:

from django.conf.urls import url from boards import views urlpatterns = [ url(r'^$', views.home, name='home'), url(r'^(?P<username>[\w.@+-]+)/$', views.user_profile, name='user_profile'), ]

This will match all valid usernames for a Django User model.

Now observe that the example above is a very permissive URL. That means it will match lot’s of URL patterns because it is defined in the root of the URL, with no prefix like /profile/<username>/. In this case, if we wanted to define a URL named /about/, we would have do define it before the username URL pattern:

from django.conf.urls import url from boards import views urlpatterns = [ url(r'^$', views.home, name='home'), url(r'^about/$', views.about, name='about'), url(r'^(?P<username>[\w.@+-]+)/$', views.user_profile, name='user_profile'), ]

If the “about” page was defined after the username URL pattern, Django would never find it, because the word “about” would match the username regex, and the view user_profile would be processed instead of the about view function.

There are some side effects to that. For example, from now on, we would have to treat “about” as a forbidden username, because if a user picked “about” as their username, this person would never see their profile page.

Sidenote: If you want to design cool URLs for user profiles, the easiest solution to avoid URL collision is by adding a prefix like /u/vitorfs/, or like Medium does /@vitorfs/, where "@" is the prefix.

If you want no prefix at all, consider using a list of forbidden names like this: github.com/shouldbee/reserved-usernames. Or another example is an application I developed when I was learning Django; I created my list at the time: github.com/vitorfs/parsifal/.

Those collisions are very common. Take GitHub for example; they have this URL to list all the repositories you are currently watching: github.com/watching. Someone registered a username on GitHub with the name "watching," so this person can't see his profile page. We can see a user with this username exists by trying this URL: github.com/watching/repositories which was supposed to list the user's repositories, like mine for example github.com/vitorfs/repositories.

The whole idea of this kind of URL routing is to create dynamic pages where part of the URL will be used as an identifier for a certain resource, that will be used to compose a page. This identifier can be an integer ID or a string for example.

Initially, we will be working with the Board ID to create a dynamic page for the Topics. Let’s read again the example I gave at the beginning of the URLs section:

url(r'^boards/(?P<pk>\d+)/$', views.board_topics, name='board_topics')

The regex \d+ will match an integer of arbitrary size. This integer will be used to retrieve the Board from the database. Now observe that we wrote the regex as (?P<pk>\d+), this is telling Django to capture the value into a keyword argument named pk.

Here is how we write a view function for it:

def board_topics(request, pk): # do something...

Because we used the (?P<pk>\d+) regex, the keyword argument in the board_topics must be named pk.

If we wanted to use any name, we could do it like this:

url(r'^boards/(\d+)/$', views.board_topics, name='board_topics')

Then the view function could be defined like this:

def board_topics(request, board_id): # do something...

Or like this:

def board_topics(request, id): # do something...

The name wouldn’t matter. But it’s a good practice to use named parameters because when we start composing bigger URLs capturing multiple IDs and variables, it will be easier to read.

Sidenote: PK or ID?

PK stands for Primary Key. It's a shortcut for accessing a model's primary key. All Django models have this attribute.

For the most cases, using the pk property is the same as id. That's because if we don't define a primary key for a model, Django will automatically create an AutoField named id, which will be its primary key.

If you defined a different primary key for a model, for example, let's say the field email is your primary key. To access it you could either use obj.email or obj.pk.

Using the URLs API

It’s time to write some code. Let’s implement the topic listing page (see Figure 1) I mentioned at the beginning of the URLs section.

First, edit the urls.py adding our new URL route:

myproject/urls.py

from django.conf.urls import url from django.contrib import admin from boards import views urlpatterns = [ url(r'^$', views.home, name='home'), url(r'^boards/(?P<pk>\d+)/$', views.board_topics, name='board_topics'), url(r'^admin/', admin.site.urls), ]

Now let’s create the view function board_topics:

boards/views.py

from django.shortcuts import render from .models import Board def home(request): # code suppressed for brevity def board_topics(request, pk): board = Board.objects.get(pk=pk) return render(request, 'topics.html', {'board': board})

In the templates folder, create a new template named topics.html:

templates/topics.html

{% load static %}<!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>{{ board.name }}</title> <link rel="stylesheet" href="{% static 'css/bootstrap.min.css' %}"> </head> <body> <div class="container"> <ol class="breadcrumb my-4"> <li class="breadcrumb-item">Boards</li> <li class="breadcrumb-item active">{{ board.name }}</li> </ol> </div> </body> </html>

Note: For now we are simply creating new HTML templates. No worries, in the following section I will show you how to create reusable templates.

Now check the URL http://127.0.0.1:8000/boards/1/ in a web browser. The result should be the following page:

Time to write some tests! Edit the tests.py file and add the following tests in the bottom of the file:

boards/tests.py

from django.core.urlresolvers import reverse from django.urls import resolve from django.test import TestCase from .views import home, board_topics from .models import Board class HomeTests(TestCase): # ... class BoardTopicsTests(TestCase): def setUp(self): Board.objects.create(name='Django', description='Django board.') def test_board_topics_view_success_status_code(self): url = reverse('board_topics', kwargs={'pk': 1}) response = self.client.get(url) self.assertEquals(response.status_code, 200) def test_board_topics_view_not_found_status_code(self): url = reverse('board_topics', kwargs={'pk': 99}) response = self.client.get(url) self.assertEquals(response.status_code, 404) def test_board_topics_url_resolves_board_topics_view(self): view = resolve('/boards/1/') self.assertEquals(view.func, board_topics)

A few things to note here. This time we used the setUp method. In the setup method, we created a Board instance, so to use it in the tests. We have to do that because the Django testing suite doesn’t run your tests against the current database. To run the tests Django creates a new database on the fly, apply all the model migrations, run the tests, and when it’s done, it destroys the testing database.

So in the setUp method, we prepare the environment to run the tests, so to simulate a scenario.

  • The test_board_topics_view_success_status_code method: is testing if Django is returning a status code 200 (success) for an existing Board.
  • The test_board_topics_view_not_found_status_code method: is testing if Django is returning a status code 404 (page not found) for a Board that doesn’t exist in the database.
  • The test_board_topics_url_resolves_board_topics_view method: is testing if Django is using the correct view function to render the topics.

Now it’s time to run the tests:

python manage.py test

And the output:

Creating test database for alias 'default'... System check identified no issues (0 silenced). .E... ====================================================================== ERROR: test_board_topics_view_not_found_status_code (boards.tests.BoardTopicsTests) ---------------------------------------------------------------------- Traceback (most recent call last): # ... boards.models.DoesNotExist: Board matching query does not exist. ---------------------------------------------------------------------- Ran 5 tests in 0.093s FAILED (errors=1) Destroying test database for alias 'default'...

The test test_board_topics_view_not_found_status_code failed. We can see in the Traceback it returned an exception “boards.models.DoesNotExist: Board matching query does not exist.”

In production with DEBUG=False, the visitor would see a 500 Internal Server Error page. But that’s not the behavior we want.

We want to show a 404 Page Not Found. So let’s refactor our view:

boards/views.py

from django.shortcuts import render from django.http import Http404 from .models import Board def home(request): # code suppressed for brevity def board_topics(request, pk): try: board = Board.objects.get(pk=pk) except Board.DoesNotExist: raise Http404 return render(request, 'topics.html', {'board': board})

Let’s test again:

python manage.py test Creating test database for alias 'default'... System check identified no issues (0 silenced). ..... ---------------------------------------------------------------------- Ran 5 tests in 0.042s OK Destroying test database for alias 'default'...

Yay! Now it’s working as expected.

This is the default page Django show while with DEBUG=False. Later on, we can customize the 404 page to show something else.

Now that’s a very common use case. In fact, Django has a shortcut to try to get an object, or return a 404 with the object does not exist.

So let’s refactor the board_topics view again:

from django.shortcuts import render, get_object_or_404 from .models import Board def home(request): # code suppressed for brevity def board_topics(request, pk): board = get_object_or_404(Board, pk=pk) return render(request, 'topics.html', {'board': board})

Changed the code? Test it.

python manage.py test Creating test database for alias 'default'... System check identified no issues (0 silenced). ..... ---------------------------------------------------------------------- Ran 5 tests in 0.052s OK Destroying test database for alias 'default'...

Didn’t break anything. We can proceed with the development.

The next step now is to create the navigation links in the screens. The homepage should have a link to take the visitor to the topics page of a given Board. Similarly, the topics page should have a link back to the homepage.

We can start by writing some tests for the HomeTests class:

boards/tests.py

class HomeTests(TestCase): def setUp(self): self.board = Board.objects.create(name='Django', description='Django board.') url = reverse('home') self.response = self.client.get(url) def test_home_view_status_code(self): self.assertEquals(self.response.status_code, 200) def test_home_url_resolves_home_view(self): view = resolve('/') self.assertEquals(view.func, home) def test_home_view_contains_link_to_topics_page(self): board_topics_url = reverse('board_topics', kwargs={'pk': self.board.pk}) self.assertContains(self.response, 'href="{0}"'.format(board_topics_url))

Observe that now we added a setUp method for the HomeTests as well. That’s because now we are going to need a Board instance and also we moved the url and response to the setUp, so we can reuse the same response in the new test.

The new test here is the test_home_view_contains_link_to_topics_page. Here we are using the assertContains method to test if the response body contains a given text. The text we are using in the test, is the href part of an a tag. So basically we are testing if the response body has the text href="/boards/1/".

Let’s run the tests:

python manage.py test Creating test database for alias 'default'... System check identified no issues (0 silenced). ....F. ====================================================================== FAIL: test_home_view_contains_link_to_topics_page (boards.tests.HomeTests) ---------------------------------------------------------------------- # ... AssertionError: False is not true : Couldn't find 'href="/boards/1/"' in response ---------------------------------------------------------------------- Ran 6 tests in 0.034s FAILED (failures=1) Destroying test database for alias 'default'...

Now we can write the code that will make this test pass.

Edit the home.html template:

templates/home.html

<!-- code suppressed for brevity --> <tbody> {% for board in boards %} <tr> <td> <a href="{% url 'board_topics' board.pk %}">{{ board.name }}</a> <small class="text-muted d-block">{{ board.description }}</small> </td> <td class="align-middle">0</td> <td class="align-middle">0</td> <td></td> </tr> {% endfor %} </tbody> <!-- code suppressed for brevity -->

So basically we changed the line:

{{ board.name }}

To:

<a href="{% url 'board_topics' board.pk %}">{{ board.name }}</a>

Always use the {% url %} template tag to compose the applications URLs. The first parameter is the name of the URL (defined in the URLconf, i.e., the urls.py), then you can pass an arbitrary number of arguments as needed.

If it were a simple URL, like the homepage, it would be just {% url 'home' %}.

Save the file and run the tests again:

python manage.py test Creating test database for alias 'default'... System check identified no issues (0 silenced). ...... ---------------------------------------------------------------------- Ran 6 tests in 0.037s OK Destroying test database for alias 'default'...

Good! Now we can check how it looks like in the web browser:

Now the link back. We can write the test first:

boards/tests.py

class BoardTopicsTests(TestCase): # code suppressed for brevity... def test_board_topics_view_contains_link_back_to_homepage(self): board_topics_url = reverse('board_topics', kwargs={'pk': 1}) response = self.client.get(board_topics_url) homepage_url = reverse('home') self.assertContains(response, 'href="{0}"'.format(homepage_url))

Run the tests:

python manage.py test Creating test database for alias 'default'... System check identified no issues (0 silenced). .F..... ====================================================================== FAIL: test_board_topics_view_contains_link_back_to_homepage (boards.tests.BoardTopicsTests) ---------------------------------------------------------------------- Traceback (most recent call last): # ... AssertionError: False is not true : Couldn't find 'href="/"' in response ---------------------------------------------------------------------- Ran 7 tests in 0.054s FAILED (failures=1) Destroying test database for alias 'default'...

Update the board topics template:

templates/topics.html

{% load static %}<!DOCTYPE html> <html> <head><!-- code suppressed for brevity --></head> <body> <div class="container"> <ol class="breadcrumb my-4"> <li class="breadcrumb-item"><a href="{% url 'home' %}">Boards</a></li> <li class="breadcrumb-item active">{{ board.name }}</li> </ol> </div> </body> </html>

Run the tests:

python manage.py test Creating test database for alias 'default'... System check identified no issues (0 silenced). ....... ---------------------------------------------------------------------- Ran 7 tests in 0.061s OK Destroying test database for alias 'default'...

As I mentioned before, URL routing is a fundamental part of a web application. With this knowledge, we should be able to proceed with the development. Next, to complete the section about URLs, you will find a summary of useful URL patterns.

List of Useful URL Patterns

The trick part is the regex. So I prepared a list of the most used URL patterns. You can always refer to this list when you need a specific URL.

Primary Key AutoField Regex (?P<pk>\d+) Example url(r'^questions/(?P<pk>\d+)/$', views.question, name='question') Valid URL /questions/934/ Captures {'pk': '934'} Slug Field Regex (?P<slug>[-\w]+) Example url(r'^posts/(?P<slug>[-\w]+)/$', views.post, name='post') Valid URL /posts/hello-world/ Captures {'slug': 'hello-world'} Slug Field with Primary Key Regex (?P<slug>[-\w]+)-(?P<pk>\d+) Example url(r'^blog/(?P<slug>[-\w]+)-(?P<pk>\d+)/$', views.blog_post, name='blog_post') Valid URL /blog/hello-world-159/ Captures {'slug': 'hello-world', 'pk': '159'} Django User Username Regex (?P<username>[\w.@+-]+) Example url(r'^profile/(?P<username>[\w.@+-]+)/$', views.user_profile, name='user_profile') Valid URL /profile/vitorfs/ Captures {'username': 'vitorfs'} Year Regex (?P<year>[0-9]{4}) Example url(r'^articles/(?P<year>[0-9]{4})/$', views.year_archive, name='year') Valid URL /articles/2016/ Captures {'year': '2016'} Year / Month Regex (?P<year>[0-9]{4})/(?P<month>[0-9]{2}) Example url(r'^articles/(?P<year>[0-9]{4})/(?P<month>[0-9]{2})/$', views.month_archive, name='month') Valid URL /articles/2016/01/ Captures {'year': '2016', 'month': '01'}

You can find more details about those patterns in this post: List of Useful URL Patterns.

Reusable Templates

Until now we’ve been copying and pasting HTML repeating several parts of the HTML document, which is not very sustainable in the long run. It’s also a bad practice.

In this section we are going to refactor our HTML templates, creating a master page and only adding the unique part for each template.

Create a new file named base.html in the templates folder:

templates/base.html

{% load static %}<!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>{% block title %}Django Boards{% endblock %}</title> <link rel="stylesheet" href="{% static 'css/bootstrap.min.css' %}"> </head> <body> <div class="container"> <ol class="breadcrumb my-4"> {% block breadcrumb %} {% endblock %} </ol> {% block content %} {% endblock %} </div> </body> </html>

This is going to be our master page. Every template we create, is going to extend this special template. Observe now we introduced the {% block %} tag. It is used to reserve a space in the template, which a “child” template (which extends the master page) can insert code and HTML within that space.

In the case of the {% block title %} we are also setting a default value, which is “Django Boards.” It will be used if we don’t set a value for the {% block title %} in a child template.

Now let’s refactor our two templates: home.html and topics.html.

templates/home.html

{% extends 'base.html' %} {% block breadcrumb %} <li class="breadcrumb-item active">Boards</li> {% endblock %} {% block content %} <table class="table"> <thead class="thead-inverse"> <tr> <th>Board</th> <th>Posts</th> <th>Topics</th> <th>Last Post</th> </tr> </thead> <tbody> {% for board in boards %} <tr> <td> <a href="{% url 'board_topics' board.pk %}">{{ board.name }}</a> <small class="text-muted d-block">{{ board.description }}</small> </td> <td class="align-middle">0</td> <td class="align-middle">0</td> <td></td> </tr> {% endfor %} </tbody> </table> {% endblock %}

The first line in the home.html template is {% extends 'base.html' %}. This tag is telling Django to use the base.html template as a master page. After that, we are using the the blocks to put the unique content of the page.

templates/topics.html

{% extends 'base.html' %} {% block title %} {{ board.name }} - {{ block.super }} {% endblock %} {% block breadcrumb %} <li class="breadcrumb-item"><a href="{% url 'home' %}">Boards</a></li> <li class="breadcrumb-item active">{{ board.name }}</li> {% endblock %} {% block content %} <!-- just leaving it empty for now. we will add core here soon. --> {% endblock %}

In the topics.html template, we are changing the {% block title %} default value. Notice that we can reuse the default value of the block by calling {{ block.super }}. So here we are playing with the website title, which we defined in the base.html as “Django Boards.” So for the “Python” board page, the title will be “Python - Django Boards,” for the “Random” board the title will be “Random - Django Boards.”

Now let’s run the tests and see we didn’t break anything:

python manage.py test Creating test database for alias 'default'... System check identified no issues (0 silenced). ....... ---------------------------------------------------------------------- Ran 7 tests in 0.067s OK Destroying test database for alias 'default'...

Great! Everything is looking good.

Now that we have the base.html template, we can easily add a top bar with a menu:

templates/base.html

{% load static %}<!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>{% block title %}Django Boards{% endblock %}</title> <link rel="stylesheet" href="{% static 'css/bootstrap.min.css' %}"> </head> <body> <nav class="navbar navbar-expand-lg navbar-dark bg-dark"> <div class="container"> <a class="navbar-brand" href="{% url 'home' %}">Django Boards</a> </div> </nav> <div class="container"> <ol class="breadcrumb my-4"> {% block breadcrumb %} {% endblock %} </ol> {% block content %} {% endblock %} </div> </body> </html>

The HTML I used is part of the Bootstrap 4 Navbar Component.

A nice touch I like to add is to change the font in the “logo” (.navbar-brand) of the page.

Go to fonts.google.com, type “Django Boards” or whatever name you gave to your project then click on apply to all fonts. Browse a bit, find one that you like.

Add the font in the base.html template:

{% load static %}<!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>{% block title %}Django Boards{% endblock %}</title> <link href="https://fonts.googleapis.com/css?family=Peralta" rel="stylesheet"> <link rel="stylesheet" href="{% static 'css/bootstrap.min.css' %}"> <link rel="stylesheet" href="{% static 'css/app.css' %}"> </head> <body> <!-- code suppressed for brevity --> </body> </html>

Now create a new CSS file named app.css inside the static/css folder:

static/css/app.css

.navbar-brand { font-family: 'Peralta', cursive; }

Forms

Forms are used to deal with user input. It’s a very common task in any web application or website. The standard way to do it is through HTML forms, where the user input some data, submit it to the server, and then the server does something with it.

Form processing is a fairly complex task because it involves interacting with many layers of an application. There are also many issues to take care. For example, all data submitted to the server comes in a string format, so we have to transform it into a proper data type (integer, float, date, etc.) before doing anything with it. We have to validate the data regarding the business logic of the application. We also have to clean, sanitize the data properly so to avoid security issues such as SQL Injection and XSS attacks.

Good news is that the Django Forms API makes the whole process a lot easier, automating a good chunk of this work. Also, the final result is a much more secure code than most programmers would be able to implement by themselves. So, no matter how simple the HTML form is, always use the forms API.

How Not Implement a Form

At first, I thought about jumping straight to the forms API. But I think it would be a good idea for us to spend some time trying to understand the underlying details of form processing. Otherwise, it will end up looking like magic, which is a bad thing, because when things go wrong, you have no idea where to look for the problem.

With a deeper understanding of some programming concepts, we can feel more in control of the situation. Being in control is important because it let us write code with more confidence. The moment we know exactly what is going on, it’s much easier to implement a code of predictable behavior. It’s also a lot easier to debug and find errors because you know where to look at.

Anyway, let’s start by implementing the form below:

It’s one of the wireframes we draw in the previous tutorial. I now realize this may be a bad example to start because this particular form involves processing data of two different models: Topic (subject) and Post (message).

There’s another important aspect that we haven’t discussed it so far, which is user authentication. We are only supposed to show this screen for authenticated users. This way we can tell who created a Topic or a Post.

So let’s abstract some details for now and focus on understanding how to save user input in the database.

First thing, let’s create a new URL route named new_topic:

myproject/urls.py

from django.conf.urls import url from django.contrib import admin from boards import views urlpatterns = [ url(r'^$', views.home, name='home'), url(r'^boards/(?P<pk>\d+)/$', views.board_topics, name='board_topics'), url(r'^boards/(?P<pk>\d+)/new/$', views.new_topic, name='new_topic'), url(r'^admin/', admin.site.urls), ]

The way we are building the URL will help us identify the correct Board.

Now let’s create the new_topic view function:

boards/views.py

from django.shortcuts import render, get_object_or_404 from .models import Board def new_topic(request, pk): board = get_object_or_404(Board, pk=pk) return render(request, 'new_topic.html', {'board': board})

For now, the new_topic view function is looking exactly the same as the board_topics. That’s on purpose, let’s take a step at a time.

Now we just need a template named new_topic.html to see some code working:

templates/new_topic.html

{% extends 'base.html' %} {% block title %}Start a New Topic{% endblock %} {% block breadcrumb %} <li class="breadcrumb-item"><a href="{% url 'home' %}">Boards</a></li> <li class="breadcrumb-item"><a href="{% url 'board_topics' board.pk %}">{{ board.name }}</a></li> <li class="breadcrumb-item active">New topic</li> {% endblock %} {% block content %} {% endblock %}

For now we just have the breadcrumb assuring the navigation. Observe that we included the URL back to the board_topics view.

Open the URL http://127.0.0.1:8000/boards/1/new/. The result, for now, is the following page:

We still haven’t implemented a way to reach this new page, but if we change the URL to http://127.0.0.1:8000/boards/2/new/, it should take us to the Python Board:

Note:

The result may be different for you if you haven't followed the steps from the previous tutorial. In my case, I have three Board instances in the database, being Django = 1, Python = 2, and Random = 3. Those numbers are the IDs from the database, used from the URL to identify the right resource.

We can already add some tests:

boards/tests.py

from django.core.urlresolvers import reverse from django.urls import resolve from django.test import TestCase from .views import home, board_topics, new_topic from .models import Board class HomeTests(TestCase): # ... class BoardTopicsTests(TestCase): # ... class NewTopicTests(TestCase): def setUp(self): Board.objects.create(name='Django', description='Django board.') def test_new_topic_view_success_status_code(self): url = reverse('new_topic', kwargs={'pk': 1}) response = self.client.get(url) self.assertEquals(response.status_code, 200) def test_new_topic_view_not_found_status_code(self): url = reverse('new_topic', kwargs={'pk': 99}) response = self.client.get(url) self.assertEquals(response.status_code, 404) def test_new_topic_url_resolves_new_topic_view(self): view = resolve('/boards/1/new/') self.assertEquals(view.func, new_topic) def test_new_topic_view_contains_link_back_to_board_topics_view(self): new_topic_url = reverse('new_topic', kwargs={'pk': 1}) board_topics_url = reverse('board_topics', kwargs={'pk': 1}) response = self.client.get(new_topic_url) self.assertContains(response, 'href="{0}"'.format(board_topics_url))

A quick summary of the tests of our new class NewTopicTests:

  • setUp: creates a Board instance to be used during the tests
  • test_new_topic_view_success_status_code: check if the request to the view is successful
  • test_new_topic_view_not_found_status_code: check if the view is raising a 404 error when the Board does not exist
  • test_new_topic_url_resolves_new_topic_view: check if the right view is being used
  • test_new_topic_view_contains_link_back_to_board_topics_view: ensure the navigation back to the list of topics

Run the tests:

python manage.py test Creating test database for alias 'default'... System check identified no issues (0 silenced). ........... ---------------------------------------------------------------------- Ran 11 tests in 0.076s OK Destroying test database for alias 'default'...

Good, now it’s time to start creating the form.

templates/new_topic.html

{% extends 'base.html' %} {% block title %}Start a New Topic{% endblock %} {% block breadcrumb %} <li class="breadcrumb-item"><a href="{% url 'home' %}">Boards</a></li> <li class="breadcrumb-item"><a href="{% url 'board_topics' board.pk %}">{{ board.name }}</a></li> <li class="breadcrumb-item active">New topic</li> {% endblock %} {% block content %} <form method="post"> {% csrf_token %} <div class="form-group"> <label for="id_subject">Subject</label> <input type="text" class="form-control" id="id_subject" name="subject"> </div> <div class="form-group"> <label for="id_message">Message</label> <textarea class="form-control" id="id_message" name="message" rows="5"></textarea> </div> <button type="submit" class="btn btn-success">Post</button> </form> {% endblock %}

This is a raw HTML form created by hand using the CSS classes provided by Bootstrap 4. It looks like this:

In the <form> tag, we have to define the method attribute. This instructs the browser on how we want to communicate with the server. The HTTP spec defines several request methods (verbs). But for the most part, we will only be using GET and POST request types.

GET is perhaps the most common request type. It’s used to retrieve data from the server. Every time you click on a link or type a URL directly into the browser, you are creating a GET request.

POST is used when we want to change data on the server. So, generally speaking, every time we send data to the server that will result in a change in the state of a resource, we should always send it via POST request.

Django protects all POST requests using a CSRF Token (Cross-Site Request Forgery Token). It’s a security measure to avoid external sites or applications to submit data to our application. Every time the application receives a POST, it will first look for the CSRF Token. If the request has no token, or the token is invalid, it will discard the posted data.

The result of the csrf_token template tag:

{% csrf_token %}

Is a hidden field that’s submitted along with the other form data:

<input type="hidden" name="csrfmiddlewaretoken" value="jG2o6aWj65YGaqzCpl0TYTg5jn6SctjzRZ9KmluifVx0IVaxlwh97YarZKs54Y32">

Another thing, we have to set the name of the HTML inputs. The name will be used to retrieve the data on the server side.

<input type="text" class="form-control" id="id_subject" name="subject"> <textarea class="form-control" id="id_message" name="message" rows="5"></textarea>

Here is how we retrieve the data:

subject = request.POST['subject'] message = request.POST['message']

So, a naïve implementation of a view that grabs the data from the HTML and start a new topic can be written like this:

from django.contrib.auth.models import User from django.shortcuts import render, redirect, get_object_or_404 from .models import Board, Topic, Post def new_topic(request, pk): board = get_object_or_404(Board, pk=pk) if request.method == 'POST': subject = request.POST['subject'] message = request.POST['message'] user = User.objects.first() # TODO: get the currently logged in user topic = Topic.objects.create( subject=subject, board=board, starter=user ) post = Post.objects.create( message=message, topic=topic, created_by=user ) return redirect('board_topics', pk=board.pk) # TODO: redirect to the created topic page return render(request, 'new_topic.html', {'board': board})

This view is only considering the happy path, which is receiving the data and saving it into the database. But there are some missing parts. We are not validating the data. The user could submit an empty form or a subject that’s bigger than 255 characters.

So far we are hard-coding the User fields because we haven’t implemented the authentication yet. But there’s an easy way to identify the logged in user. We will get to that part in the next tutorial. Also, we haven’t implemented the view where we will list all the posts within a topic, so upon success, we are redirecting the user to the page where we list all the board topics.

Submitted the form clicking on the Post button:

It looks like it worked. But we haven’t implemented the topics listing yet, so there’s nothing to see here. Let’s edit the templates/topics.html file to do a proper listing:

templates/topics.html

{% extends 'base.html' %} {% block title %} {{ board.name }} - {{ block.super }} {% endblock %} {% block breadcrumb %} <li class="breadcrumb-item"><a href="{% url 'home' %}">Boards</a></li> <li class="breadcrumb-item active">{{ board.name }}</li> {% endblock %} {% block content %} <table class="table"> <thead class="thead-inverse"> <tr> <th>Topic</th> <th>Starter</th> <th>Replies</th> <th>Views</th> <th>Last Update</th> </tr> </thead> <tbody> {% for topic in board.topics.all %} <tr> <td>{{ topic.subject }}</td> <td>{{ topic.starter.username }}</td> <td>0</td> <td>0</td> <td>{{ topic.last_updated }}</td> </tr> {% endfor %} </tbody> </table> {% endblock %}

Yep! The Topic we created is here.

Two new concepts here:

We are using for the first time the topics property in the Board model. The topics property is created automatically by Django using a reverse relationship. In the previous steps, we created a Topic instance:

def new_topic(request, pk): board = get_object_or_404(Board, pk=pk) # ... topic = Topic.objects.create( subject=subject, board=board, starter=user )

In the line board=board, we set the board field in Topic model, which is a ForeignKey(Board). With that, now our Board instance is aware that it has an Topic instance associated with it.

The reason why we used board.topics.all instead of just board.topics is because board.topics is a Related Manager, which is pretty much similar to a Model Manager, usually available on the board.objects property. So, to return all topics associated with a given board, we have to run board.topics.all(). To filter some data, we could do board.topics.filter(subject__contains='Hello').

Another important thing to note is that, inside a Python code, we have to use parenthesis: board.topics.all(), because all() is a method. When writing code using the Django Template Language, in an HTML template file, we don’t use parenthesis, so it’s just board.topics.all.

The second thing is that we are making use of a ForeignKey:

{{ topic.starter.username }}

Just create a path through the property using dots. We can pretty much access any property of the User model. If we wanted the user’s email, we could use topic.starter.email.

Since we are already modifying the topics.html template, let’s create the button that takes us to the new topic screen:

templates/topics.html

{% block content %} <div class="mb-4"> <a href="{% url 'new_topic' board.pk %}" class="btn btn-primary">New topic</a> </div> <table class="table"> <!-- code suppressed for brevity --> </table> {% endblock %}

We can include a test to make sure the user can reach the New topic view from this page:

boards/tests.py

class BoardTopicsTests(TestCase): # ... def test_board_topics_view_contains_navigation_links(self): board_topics_url = reverse('board_topics', kwargs={'pk': 1}) homepage_url = reverse('home') new_topic_url = reverse('new_topic', kwargs={'pk': 1}) response = self.client.get(board_topics_url) self.assertContains(response, 'href="{0}"'.format(homepage_url)) self.assertContains(response, 'href="{0}"'.format(new_topic_url))

Basically here I renamed the old test_board_topics_view_contains_link_back_to_homepage method and add an extra assertContains. This test is now responsible for making sure our view contains the required navigation links.

Testing The Form View

Before we code the previous form example in a Django way, let’s write some tests for the form processing:

boards/tests.py

class NewTopicTests(TestCase): def setUp(self): Board.objects.create(name='Django', description='Django board.') User.objects.create_user(username='john', email='john@doe.com', password='123') # <- included this line here # ... def test_csrf(self): url = reverse('new_topic', kwargs={'pk': 1}) response = self.client.get(url) self.assertContains(response, 'csrfmiddlewaretoken') def test_new_topic_valid_post_data(self): url = reverse('new_topic', kwargs={'pk': 1}) data = { 'subject': 'Test title', 'message': 'Lorem ipsum dolor sit amet' } response = self.client.post(url, data) self.assertTrue(Topic.objects.exists()) self.assertTrue(Post.objects.exists()) def test_new_topic_invalid_post_data(self): ''' Invalid post data should not redirect The expected behavior is to show the form again with validation errors ''' url = reverse('new_topic', kwargs={'pk': 1}) response = self.client.post(url, {}) self.assertEquals(response.status_code, 200) def test_new_topic_invalid_post_data_empty_fields(self): ''' Invalid post data should not redirect The expected behavior is to show the form again with validation errors ''' url = reverse('new_topic', kwargs={'pk': 1}) data = { 'subject': '', 'message': '' } response = self.client.post(url, data) self.assertEquals(response.status_code, 200) self.assertFalse(Topic.objects.exists()) self.assertFalse(Post.objects.exists())

First thing, the tests.py file is already starting to get big. We will improve it soon, breaking the tests into several files. But for now, let’s keep working on it.

  • setUp: included the User.objects.create_user to create a User instance to be used in the tests
  • test_csrf: since the CSRF Token is a fundamental part of processing POST requests, we have to make sure our HTML contains the token.
  • test_new_topic_valid_post_data: sends a valid combination of data and check if the view created a Topic instance and a Post instance.
  • test_new_topic_invalid_post_data: here we are sending an empty dictionary to check how the application is behaving.
  • test_new_topic_invalid_post_data_empty_fields: similar to the previous test, but this time we are sending some data. The application is expected to validate and reject empty subject and message.

Let’s run the tests:

python manage.py test Creating test database for alias 'default'... System check identified no issues (0 silenced). ........EF..... ====================================================================== ERROR: test_new_topic_invalid_post_data (boards.tests.NewTopicTests) ---------------------------------------------------------------------- Traceback (most recent call last): ... django.utils.datastructures.MultiValueDictKeyError: "'subject'" ====================================================================== FAIL: test_new_topic_invalid_post_data_empty_fields (boards.tests.NewTopicTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/vitorfs/Development/myproject/django-beginners-guide/boards/tests.py", line 115, in test_new_topic_invalid_post_data_empty_fields self.assertEquals(response.status_code, 200) AssertionError: 302 != 200 ---------------------------------------------------------------------- Ran 15 tests in 0.512s FAILED (failures=1, errors=1) Destroying test database for alias 'default'...

We have one failing test and one error. Both related to invalid user input. Instead of trying to fix it with the current implementation, let’s make those tests pass using the Django Forms API.

Creating Forms The Right Way

So, we came a long way since we started working with Forms. Finally, it’s time to use the Forms API.

The Forms API is available in the module django.forms. Django works with two types of forms: forms.Form and forms.ModelForm. The Form class is a general purpose form implementation. We can use it to process data that are not directly associated with a model in our application. A ModelForm is a subclass of Form, and it’s associated with a model class.

Let’s create a new file named forms.py inside the boards’ folder:

boards/forms.py

from django import forms from .models import Topic class NewTopicForm(forms.ModelForm): message = forms.CharField(widget=forms.Textarea(), max_length=4000) class Meta: model = Topic fields = ['subject', 'message']

This is our first form. It’s a ModelForm associated with the Topic model. The subject in the fields list inside the Meta class is referring to the subject field in the Topic class. Now observe that we are defining an extra field named message. This refers to the message in the Post we want to save.

Now we have to refactor our views.py:

boards/views.py

from django.contrib.auth.models import User from django.shortcuts import render, redirect, get_object_or_404 from .forms import NewTopicForm from .models import Board, Topic, Post def new_topic(request, pk): board = get_object_or_404(Board, pk=pk) user = User.objects.first() # TODO: get the currently logged in user if request.method == 'POST': form = NewTopicForm(request.POST) if form.is_valid(): topic = form.save(commit=False) topic.board = board topic.starter = user topic.save() post = Post.objects.create( message=form.cleaned_data.get('message'), topic=topic, created_by=user ) return redirect('board_topics', pk=board.pk) # TODO: redirect to the created topic page else: form = NewTopicForm() return render(request, 'new_topic.html', {'board': board, 'form': form})

This is how we use the forms in a view. Let me remove the extra noise so we can focus on the core of the form processing:

if request.method == 'POST': form = NewTopicForm(request.POST) if form.is_valid(): topic = form.save() return redirect('board_topics', pk=board.pk) else: form = NewTopicForm() return render(request, 'new_topic.html', {'form': form})

First we check if the request is a POST or a GET. If the request came from a POST, it means the user is submitting some data to the server. So we instantiate a form instance passing the POST data to the form: form = NewTopicForm(request.POST).

Then, we ask Django to verify the data, check if the form is valid if we can save it in the database: if form.is_valid():. If the form was valid, we proceed to save the data in the database using form.save(). The save() method returns an instance of the Model saved into the database. So, since this is a Topic form, it will return the Topic that was created: topic = form.save(). After that, the common path is to redirect the user somewhere else, both to avoid the user re-submit the form by pressing F5 and also to keep the flow of the application.

Now, if the data was invalid, Django will add a list of errors to the form. After that, the view does nothing and return in the last statement: return render(request, 'new_topic.html', {'form': form}). That means we have to update the new_topic.html to display errors properly.

If the request was a GET, we just initialize a new and empty form using form = NewTopicForm().

Let’s run the tests and see how is everything:

python manage.py test Creating test database for alias 'default'... System check identified no issues (0 silenced). ............... ---------------------------------------------------------------------- Ran 15 tests in 0.522s OK Destroying test database for alias 'default'...

We even fixed the last two tests.

The Django Forms API does much more than processing and validating the data. It also generates the HTML for us.

Let’s update the new_topic.html template to fully use the Django Forms API:

templates/new_topic.html

{% extends 'base.html' %} {% block title %}Start a New Topic{% endblock %} {% block breadcrumb %} <li class="breadcrumb-item"><a href="{% url 'home' %}">Boards</a></li> <li class="breadcrumb-item"><a href="{% url 'board_topics' board.pk %}">{{ board.name }}</a></li> <li class="breadcrumb-item active">New topic</li> {% endblock %} {% block content %} <form method="post"> {% csrf_token %} {{ form.as_p }} <button type="submit" class="btn btn-success">Post</button> </form> {% endblock %}

The form have three rendering options: form.as_table, form.as_ul, and form.as_p. It’s a quick way to render all the fields of a form. As the name suggests, the as_table uses table tags to format the inputs, the as_ul creates an HTML list of inputs, etc.

Let’s see how it looks like:

Well, our previous form was looking better, right? We are going to fix it in a moment.

It can look broken right now but trust me; there’s a lot of things behind it right now. And it’s extremely powerful. For example, if our form had 50 fields, we could render all the fields just by typing {{ form.as_p }}.

And more, using the Forms API, Django will validate the data and add error messages to each field. Let’s try submitting an empty form:

Note:

If you see something like this: when you submit the form, that's not Django. It's your browser doing a pre-validation. To disable it add the novalidate attribute to your form tag: <form method="post" novalidate>

You can keep it; there's no problem with it. It's just because our form is very simple right now, and we don't have much data validation to see.

Another important thing to note is that: there is no such a thing as "client-side validation." JavaScript validation or browser validation is just for usability purpose. And also to reduce the number of requests to the server. Data validation should always be done on the server side, where we have full control over the data.

It also handles help texts, which can be defined both in a Form class or in a Model class:

boards/forms.py

from django import forms from .models import Topic class NewTopicForm(forms.ModelForm): message = forms.CharField( widget=forms.Textarea(), max_length=4000, help_text='The max length of the text is 4000.' ) class Meta: model = Topic fields = ['subject', 'message']

We can also set extra attributes to a form field:

boards/forms.py

from django import forms from .models import Topic class NewTopicForm(forms.ModelForm): message = forms.CharField( widget=forms.Textarea( attrs={'rows': 5, 'placeholder': 'What is in your mind?'} ), max_length=4000, help_text='The max length of the text is 4000.' ) class Meta: model = Topic fields = ['subject', 'message']

Rendering Bootstrap Forms

Alright, so let’s make things pretty again.

When working with Bootstrap or any other Front-End library, I like to use a Django package called django-widget-tweaks. It gives us more control over the rendering process, keeping the defaults and just adding extra customizations on top of it.

Let’s start off by installing it:

pip install django-widget-tweaks

Now add it to the INSTALLED_APPS:

myproject/settings.py

INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'widget_tweaks', 'boards', ]

Now let’s take it into use:

templates/new_topic.html

{% extends 'base.html' %} {% load widget_tweaks %} {% block title %}Start a New Topic{% endblock %} {% block breadcrumb %} <li class="breadcrumb-item"><a href="{% url 'home' %}">Boards</a></li> <li class="breadcrumb-item"><a href="{% url 'board_topics' board.pk %}">{{ board.name }}</a></li> <li class="breadcrumb-item active">New topic</li> {% endblock %} {% block content %} <form method="post" novalidate> {% csrf_token %} {% for field in form %} <div class="form-group"> {{ field.label_tag }} {% render_field field class="form-control" %} {% if field.help_text %} <small class="form-text text-muted"> {{ field.help_text }} </small> {% endif %} </div> {% endfor %} <button type="submit" class="btn btn-success">Post</button> </form> {% endblock %}

There it is! So, here we are using the django-widget-tweaks. First, we load it in the template by using the {% load widget_tweaks %} template tag. Then the usage:

{% render_field field class="form-control" %}

The render_field tag is not part of Django; it lives inside the package we installed. To use it we have to pass a form field instance as the first parameter, and then after we can add arbitrary HTML attributes to complement it. It will be useful because then we can assign classes based on certain conditions.

Some examples of the render_field template tag:

{% render_field form.subject class="form-control" %} {% render_field form.message class="form-control" placeholder=form.message.label %} {% render_field field class="form-control" placeholder="Write a message!" %} {% render_field field style="font-size: 20px" %}

Now to implement the Bootstrap 4 validation tags, we can change the new_topic.html template:

templates/new_topic.html

<form method="post" novalidate> {% csrf_token %} {% for field in form %} <div class="form-group"> {{ field.label_tag }} {% if form.is_bound %} {% if field.errors %} {% render_field field class="form-control is-invalid" %} {% for error in field.errors %} <div class="invalid-feedback"> {{ error }} </div> {% endfor %} {% else %} {% render_field field class="form-control is-valid" %} {% endif %} {% else %} {% render_field field class="form-control" %} {% endif %} {% if field.help_text %} <small class="form-text text-muted"> {{ field.help_text }} </small> {% endif %} </div> {% endfor %} <button type="submit" class="btn btn-success">Post</button> </form>

The result is this:

So, we have three different rendering states:

  • Initial state: the form has no data (is not bound)
  • Invalid: we add the .is-invalid CSS class and add error messages in an element with a class .invalid-feedback. The form field and the messages are rendered in red.
  • Valid: we add the .is-valid CSS class so to paint the form field in green, giving feedback to the user that this field is good to go.
Reusable Forms Templates

The template code looks a little bit complicated, right? Well, the good news is that we can reuse this snippet across the project.

In the templates folder, create a new folder named includes:

myproject/ |-- myproject/ | |-- boards/ | |-- myproject/ | |-- templates/ | | |-- includes/ <-- here! | | |-- base.html | | |-- home.html | | |-- new_topic.html | | +-- topics.html | +-- manage.py +-- venv/

Now inside the includes folder, create a file named form.html:

templates/includes/form.html

{% load widget_tweaks %} {% for field in form %} <div class="form-group"> {{ field.label_tag }} {% if form.is_bound %} {% if field.errors %} {% render_field field class="form-control is-invalid" %} {% for error in field.errors %} <div class="invalid-feedback"> {{ error }} </div> {% endfor %} {% else %} {% render_field field class="form-control is-valid" %} {% endif %} {% else %} {% render_field field class="form-control" %} {% endif %} {% if field.help_text %} <small class="form-text text-muted"> {{ field.help_text }} </small> {% endif %} </div> {% endfor %}

Now we change our new_topic.html template:

templates/new_topic.html

{% extends 'base.html' %} {% block title %}Start a New Topic{% endblock %} {% block breadcrumb %} <li class="breadcrumb-item"><a href="{% url 'home' %}">Boards</a></li> <li class="breadcrumb-item"><a href="{% url 'board_topics' board.pk %}">{{ board.name }}</a></li> <li class="breadcrumb-item active">New topic</li> {% endblock %} {% block content %} <form method="post" novalidate> {% csrf_token %} {% include 'includes/form.html' %} <button type="submit" class="btn btn-success">Post</button> </form> {% endblock %}

As the name suggests, the {% include %} is used to include HTML templates in another template. It’s a very useful way to reuse HTML components in a project.

The next form we implement, we can simply use {% include 'includes/form.html' %} to render it.

Adding More Tests

Now we are using Django Forms; we can add more tests to make sure it is running smoothly:

boards/tests.py

# ... other imports from .forms import NewTopicForm class NewTopicTests(TestCase): # ... other tests def test_contains_form(self): # <- new test url = reverse('new_topic', kwargs={'pk': 1}) response = self.client.get(url) form = response.context.get('form') self.assertIsInstance(form, NewTopicForm) def test_new_topic_invalid_post_data(self): # <- updated this one ''' Invalid post data should not redirect The expected behavior is to show the form again with validation errors ''' url = reverse('new_topic', kwargs={'pk': 1}) response = self.client.post(url, {}) form = response.context.get('form') self.assertEquals(response.status_code, 200) self.assertTrue(form.errors)

Now we are using the assertIsInstance method for the first time. Basically we are grabbing the form instance in the context data, and checking if it is a NewTopicForm. In the last test, we added the self.assertTrue(form.errors) to make sure the form is showing errors when the data is invalid.

Conclusions

In this tutorial, we focused on URLs, Reusable Templates, and Forms. As usual, we also implement several test cases. That’s how we develop with confidence.

Our tests file is starting to get big, so in the next tutorial, we are going to refactor it to improve the maintainability so to sustain the growth of our code base.

We are also reaching a point where we need to interact with the logged in user. In the next tutorial, we are going to learn everything about authentication and how to protect our views and resources.

I hope you enjoyed the third part of this tutorial series! The forth part is coming out next week, on Sep 25, 2017. If you would like to get notified when the forth part is out, you can subscribe to our mailing list.

The source code of the project is available on GitHub. The current state of the project can be found under the release tag v0.3-lw. The link below will take you to the right place:

https://github.com/sibtc/django-beginners-guide/tree/v0.3-lw

← Part 2 - Fundamentals Part 4 - Authentication (coming soon) Subscribe to our mailing list to get notified when it's out.
Categories: FLOSS Project Planets

New Kate on Windows installer

Planet KDE - Sun, 2017-09-17 16:28

We have a new round of Kate installers for Windows. This round comes with some installer-bug fixes (all project plugin dlls) and support for editorconfig files http://editorconfig.org/.

In the above screen-shot you can see the project plugin analyzing Kate code. This works if you have cppcheck installed and added to the path.

Grab the installers now at download.kde.org:  Kate-setup-17.08.1-KF5.38-32bit or Kate-setup-17.08.1-KF5.38-64bit

Categories: FLOSS Project Planets

Russ Allbery: Free software log (July and August 2017)

Planet Debian - Sun, 2017-09-17 16:08

I've wanted to start making one of these posts for a few months but have struggled to find the time. But it seems like a good idea, particularly since I get more done when I write down what I do, so you all get a rather belated one. This covers July and August; hopefully the September one will come closer to the end of September.

Debian

August was DebConf, which included a ton of Policy work thanks to Sean Whitton's energy and encouragement. During DebConf, we incorporated work from Hideki Yamane to convert Policy to reStructuredText, which has already made it far easier to maintain. (Thanks also to David Bremner for a lot of proofreading of the result.) We also did a massive bug triage and closed a ton of older bugs on which there had been no forward progress for many years.

After DebConf, as expected, we flushed out various bugs in the reStructuredText conversion and build infrastructure. I fixed a variety of build and packaging issues and started doing some more formatting cleanup, including moving some footnotes to make the resulting document more readable.

During July and August, partly at DebConf and partly not, I also merged wording fixes for seven bugs and proposed wording (not yet finished) for three more, as well as participated in various Policy discussions.

Policy was nearly all of my Debian work over these two months, but I did upload a new version of the webauth package to build with OpenSSL 1.1 and drop transitional packages.

Kerberos

I still haven't decided my long-term strategy with the Kerberos packages I maintain. My personal use of Kerberos is now fairly marginal, but I still care a lot about the software and can't convince myself to give it up.

This month, I started dusting off pam-krb5 in preparation for a new release. There's been an open issue for a while around defer_pwchange support in Heimdal, and I spent some time on that and tracked it down to an upstream bug in Heimdal as well as a few issues in pam-krb5. The pam-krb5 issues are now fixed in Git, but I haven't gotten any response upstream from the Heimdal bug report. I also dusted off three old Heimdal patches and submitted them as upstream merge requests and reported some more deficiencies I found in FAST support. On the pam-krb5 front, I updated the test suite for the current version of Heimdal (which changed some of the prompting) and updated the portability support code, but haven't yet pulled the trigger on a new release.

Other Software

I merged a couple of pull requests in podlators, one to fix various typos (thanks, Jakub Wilk) and one to change the formatting of man page references and function names to match the current Linux manual page standard (thanks, Guillem Jover). I also documented a bad interaction with line-buffered output in the Term::ANSIColor man page. Neither of these have seen a new release yet.

Categories: FLOSS Project Planets

texinfo @ Savannah: Texinfo 6.5 released

GNU Planet! - Sun, 2017-09-17 15:43

We have released version 6.5 of Texinfo, the GNU documentation format.

It's available via a mirror (xz is much smaller than gz, but gz is available too just in case):

https://ftp.gnu.org/gnu/texinfo/texinfo-6.5.tar.xz
https://ftp.gnu.org/gnu/texinfo/texinfo-6.5.tar.gz

Please send any comments to bug-texinfo@gnu.org.

Full announcement:

http://lists.gnu.org/archive/html/info-gnu/2017-09/msg00007.html

Categories: FLOSS Project Planets

Bryan Pendleton: News of the weird, part 3

Planet Apache - Sun, 2017-09-17 15:30

This one, for a change of pace, does not come out of the pages of Wired.

But it's just as weird.

So let's turn the microphone over to the great chess blogger Dana Mackenzie: Scandal Ruins World Cup’s Best Day

everybody is talking about the stupid dispute that caused the Canadian player, Anton Kovalyov, to forfeit his game and withdraw from the tournament — all over a pair of shorts.

Probably most of my readers are already familiar with the sad details, but for those who haven’t heard yet, these seem to be the facts:

  • Kovalyov showed up for his game against Maxim Rodshtein wearing a pair of shorts. He had worn the same shorts for his previous four games. Yes, apparently he only packed this one pair of shorts for a potentially month-long chess tournament. Cue jokes about chess players’ dressing habits.
  • The chief arbiter spoke to him and told him that the players’ dress code (which is in a legal contract they sign before the tournament) requires more dignified wear. He told him to go back to his room and change.
  • Kovalyov went back to his room but never reappeared. His opponent played one move (1. d4) and won by forfeit.

Even from these facts, it seems to me that the FIDE approach was very heavy-handed. From a legal point of view it seems to me that they have greatly weakened their case by allowing Kovalyov to play four games (!) in the offending garment. The arbiter said that nobody noticed earlier. Come on! If it’s a rule, then enforce it from the beginning. If it’s not enforced, then it’s not really a rule.

Kovalyov is actually Ukrainian, playing as a Canadian citizen, but living in Brownsville, Texas, where he studies computer science and got a chess scholarship!.

Kovalyov later wrote about this on his Facebook page, then tried to delete what he wrote, then tried to close his Facebook account, then re-opened his Facebook account, then wrote about it some more.

More at The Guardian, where we find that the REAL issue may have involved an ethnic slur:

Azmaiparashvili refused to back down, said Kovalyov. “At this point I was really angry but tried not to do anything stupid, and asked him why he was so rude to me, and he said because I’m a gypsy,” he said.

He continued: “So imagine this, the round is about to start, I’m being bullied by the organiser of the tournament, being assured that I will be punished by FIDE, yelled at and racially insulted. What would you do in my situation? I think many people would have punched this person in the face or at least insulted him. I decided to leave.”

Assuming that is what actually happened, it's a shame, but clearly he made the right decision.

The internets took to calling this "shortsgate" for a little while.

But it has now passed from public interest.

Categories: FLOSS Project Planets

Dirk Eddelbuettel: RcppClassic 0.9.7

Planet Debian - Sun, 2017-09-17 15:28

A rather boring and otherwise uneventful release 0.9.7 of RcppClassic is now at CRAN. This package provides a maintained version of the otherwise deprecated first Rcpp API; no new projects should use it.

Once again no changes in user-facing code. But this makes it the first package to use the very new and shiny pinp package as the backend for its vignette, now converted to Markdown---see here for this new version. We also updated three sources files for tabs versus spaces as the current g++ version complained (correctly !!) about misleading indents. Otherwise a file src/init.c was added for dynamic registration, the Travis CI runner script was updated to using run.sh from our r-travis fork, and we now strip the library after they have been built. Again, no user code changes.

And no iterate: nobody should use this package. Rcpp is so much better in so many ways---this one is simply available as we (quite strongly) believe that APIs are contracts, and as such we hold up our end of the deal.

Courtesy of CRANberries, there are changes relative to the previous release.

Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets
Syndicate content