FLOSS Project Planets

PyCharm: Interview: Quazi Naiful on Python web frameworks and GraphQL

Planet Python - Tue, 2017-10-17 11:00

Web development has long been a key part of Python’s appeal. In modern web apps, rich frontends work with Python backends over REST. But even that’s getting re-examined, with recent interest in GraphQL as a query language for web data.

Want an introduction to what GraphQL means for Python? We’re fortunate to have Quazi Nafiul as the presenter for the “GraphQL in the Python World” October 31st webinar on October 31st. Quazi is known in PyCharm as the author of Mastering PyCharm, is a long-time Python web developer, and currently a software engineer at Suitsupply.

To set the scene for the webinar, we asked Quazi to join us for a quick interview.

Thanks for joining us on the webinar. Let’s jump right into GraphQL. What is one killer feature about it that you really like?

Declarative data fetching. This feature allows you to get exactly what you ask for and makes versioning much easier since one simply adds more functionality through extending a GraphQL model (very similar to a resource).

Give us a little background on yourself, your Python origin story and what you’re doing now.

My university was predominantly C++ and Java, and those languages although powerful, never felt fast to develop with for me. So, I was trying to look for something different and stumbled onto MITx’s Introduction to Computer Science, which taught the course in Python. I’ve been using Python ever since. After the MITx course, I started hanging out on StackOverflow, and realized how little I knew. I tried to expand my understanding by answering questions on the site.

In the meantime, I started doing consultancy work for companies in NY and London. I also was a developer evangelist for a short time. I wrote plenty of bad code back then. I also started writing a book on PyCharm around this period. Once the book was complete and published, and I got bored with remote work, I started applying to companies around the world and eventually ended up in Amsterdam as a software developer.

Similar question: what did you start with for Python web frameworks, what have you used, and what’s your preference now?

I started with Django, quickly moved to Flask and now I use both Django and Flask in my day job. I remember at the time Django had the most comprehensive beginner documentation you could find. I didn’t understand a lot of it when I was first starting out, so every part of the tutorial would send me down a rabbit hole. Eventually, I was able to understand how HTTP as a protocol works. Funnily enough, I thought that one always interacts with a database with an ORM, and as a result, my understanding of SQL was a little hampered in the beginning, but I eventually managed to fix that.

Just to close out on Python web frameworks, what do you think is “next big thing”?

I honestly don’t know. Python web frameworks market shares seem to be in a kind of equilibrium at the moment. Flask serves you well if you need to build something small to medium-sized, but indeed you can build huge applications with it. Django is amazing if you just want to get up and running with a CRUD application that sits on top of a regular RDBMS. I think Django will continue to have a strong presence with the resurgence of SQL and SQL databases. Pyramid is great if you need something super flexible, and you can hack to its very core, but it does not have ecosystems as strong as Flask or Django.

Back to GraphQL. What kinds of things will you be showing, and what should people expect to learn in the webinar?

My aims are twofold. The first thing is to introduce GraphQL, and really just show people how powerful it is, and how its features can bring tangible benefits to your team and allow your businesses to move forward. The second part of that is to show that you can use all the power that GraphQL brings along with the frameworks and libraries that you’re already used to, like Flask, Django, and SQLAlchemy.

Do you plan to have any working code in advance that people can review?

Yes, absolutely. I want people to be able to follow what I’m doing, and also take that code, and work on their own implementations and ideas. I’m going to be building a generic application, but I’ll do my best to showcase the most important features, so that they can take that code and start playing around on their own.

Does GraphQL mean throwing out your current database?

Nope. You don’t even have to throw out your current ORM (in most cases). If you’re using Django or SQLAlchemy, then you can just re-use your existing models! Although, it does not have the best support for Mongo or other NoSQL databases, and so you might need to build integrations with those databases.

Every “next big thing” has some rough edges. What are tricky parts for GraphQL in Python?

I would say having a standardized form of authentication. GraphQL implements a level-based authentication system (or rather you can implement it). I would say that a lot of different services that are used to HTTP REST, might not really like this, since clients don’t exist for all languages. One example would be Salesforce APEX and .NET.

Categories: FLOSS Project Planets

Drupal Modules: The One Percent: Drupal Modules: The One Percent —Timelogin (video tutorial)

Planet Drupal - Tue, 2017-10-17 09:47
Drupal Modules: The One Percent —Timelogin (video tutorial) NonProfit Tue, 10/17/2017 - 08:47 Episode 40

Here is where we seek to bring awareness to Drupal modules running on less than 1% of reporting sites. Today we'll look at Timelogin, a module which restricts users, based on role, from logging in during certain times of the day.

Categories: FLOSS Project Planets

The Digital Cat: A game of tokens: solution - Part 2

Planet Python - Tue, 2017-10-17 08:00

Here we are, with my solution to the second part of "A Game of Tokens", aka how to write a simple interpreter in Python and survive to tell the tale. The second part of the challenge can be found here. Again, it is never repeated too many times, this is my personal solution. Your solution can be different, and if all the test pass it is correct!

You can find the code for this part in this repository. The branch called part2 contains all the commits explained in this post, and every commit contains both the test(s) and the code that makes the test(s) pass.

Level 8 - Multiplication and division Lexer

The tests we added for the lexer already pass. This is not surprising, as the lexer is designed to return everything it doesn't know as a LITERAL (smallcalc/calc_lexer.py:119). As we already instructed the lexer to skip spaces the new operators are happily digested. I decided for this project not to assign operators a specific token, so from this point of view our lexer is pretty open and could already understand instructions like 3 $ 5 or 7 : 9, even though they do not have a mathematical meaning.


The parser is not so merciful, and the two new tests do not pass. We are explicitly calling a parse_term() method that is not defined, so a success would have been very worrying. In these two tests parse_term() is called explicitly and there is no relationship with the other parse_* methods, so we can implement it as a stand-alone processing.

We know that a term is an operation between two integers, so we can follow what we did with parse_expression(). The first thing we do is to parse the first integer, then we peek the next token and we decide what to do. If the token is a LITERAL we suppose it is the operation, otherwise we probably hit the end of the file and we will just return the previously read integer. The second element may be a simple integer or another multiplication or division, so we recursively call parse_term() and return a BinaryNode with the result.

[Note: I notice that the parse_addsymbol() could be now named parse_literal() but this wasn't done when I prepared the source code. Regardless of the name, however, what this method does is to just pack a literal in a LiteralNode and return it.]

The whole parser is now the following

class CalcParser: def __init__(self): self.lexer = clex.CalcLexer() def parse_addsymbol(self): t = self.lexer.get_token() return LiteralNode(t.value) def parse_integer(self): t = self.lexer.get_token() return IntegerNode(t.value) def parse_term(self): left = self.parse_integer() next_token = self.lexer.peek_token() if next_token.type == clex.LITERAL: operator = self.parse_addsymbol() right = self.parse_term() return BinaryNode(left, operator, right) else: return IntegerNode(left.value) def parse_expression(self): left = self.parse_integer() next_token = self.lexer.peek_token() if next_token.type == clex.LITERAL: operator = self.parse_addsymbol() right = self.parse_expression() return BinaryNode(left, operator, right) else: return IntegerNode(left.value) Visitor

The visitor was instructed to deal with sums and subtractions, and it treats everything is not the former as the latter. This is why the new tests give as results 1 and 7. We just need to extend the if statement to include the new operations

class CalcVisitor: def visit(self, node): if node['type'] == 'integer': return node['value'], node['type'] if node['type'] == 'binary': lvalue, ltype = self.visit(node['left']) rvalue, rtype = self.visit(node['right']) operator = node['operator']['value'] if operator == '+': return lvalue + rvalue, ltype elif operator == '-': return lvalue - rvalue, ltype elif operator == '*': return lvalue * rvalue, ltype elif operator == '/': return lvalue // rvalue, ltype

Now we have a pretty simple but working calculator! Enjoy the cli.py, as YOU did it this time! I rememberI was pretty excited the first time I run a command line calculator done by me. But hold tight, because you are going to learn and implement much more!

Level 9 - Mixing operators

Ouch! It looks like putting multiplications and sums in the same line is not really working. As you may recall we didn't link parse_term() with the other methods, and we use a generic function to treat literals. This works in principle, but doesn't consider operator precedence.

The current output of the parser is

{ "left": { "type": "integer", "value": 2 }, "operator": { "type": "literal", "value": "*" }, "right": { "left": { "type": "integer", "value": 3 }, "operator": { "type": "literal", "value": "+" }, "right": { "type": "integer", "value": 4 }, "type": "binary" }, "type": "binary" }

As you can clearly see the parser recognised the multiplication operator, but then returns a nested sum (the oputput of a recursive call of parse_term()). This gives the sum a greater precedence that that of the sum, which is against the mathematical rules we want to follow here. 2 * 3 + 4 shall be considered (2 * 3) + 4 and not 2 + (3 * 4).

To fix this we have to rework parse_term(). First of all it shall accept only the * and / operators, then it shall return the left part if it finds a different literal. Even parse_expression() shall change a bit: the first thing to do is to call parse_term() instead of parse_integer() and then to return the left part.

The new code is then

def parse_term(self): left = self.parse_integer() next_token = self.lexer.peek_token() if next_token.type == clex.LITERAL and next_token.value in ['*', '/']: operator = self.parse_addsymbol() right = self.parse_term() return BinaryNode(left, operator, right) return left def parse_expression(self): left = self.parse_term() next_token = self.lexer.peek_token() if next_token.type == clex.LITERAL: operator = self.parse_addsymbol() right = self.parse_expression() return BinaryNode(left, operator, right) return left

Let's see what happens parsing 2 * 3 + 4. The test calls parse_expression() which tries immediately to run parse_term(). This latter recognises 2 and *, so it calls itself recursively just before the 3 and returns the binary node. This means that the multiplication is the first operation we return, the one with higher precedence. The recursive call recognises 3 but then doesn't know what to do with + as we specifically consider only * and /, so it just returns the integer value. Back to parse_expression(), then the variable left will contain the binary node that represents 2 * 3. The function will then finish adding the binary node for the sum.

Take your time to understand the mechanism, perhaps trying with different operations like 2 + 4 * 6 - 8, which should return 18.

Level 10 - Parentheses

Let's have some Lisp time here and introduce parenthesis. As happened for the new mathematical operators, parenthesis are already accepted by the lexer as simple literals, so the first test passes without any change in the code. The parser complains, however, as it always expects an integer (smallcalc/calc_parser.py:76).

As I suggested in the post my idea is to introduce a method that parses a so-called factor, which can either be an integer of an expression between parenthesis.

class CalcParser: def __init__(self): self.lexer = clex.CalcLexer() def parse_addsymbol(self): t = self.lexer.get_token() return LiteralNode(t.value) def parse_integer(self): t = self.lexer.get_token() return IntegerNode(t.value) def parse_factor(self): next_token = self.lexer.peek_token() if next_token.type == clex.LITERAL and next_token.value == '(': self.lexer.get_token() expression = self.parse_expression() self.lexer.get_token() return expression return self.parse_integer()

Then parse_term() method now has to call parse_factor()

def parse_term(self): left = self.parse_factor() next_token = self.lexer.peek_token() if next_token.type == clex.LITERAL and next_token.value in ['*', '/']: operator = self.parse_addsymbol() right = self.parse_term() return BinaryNode(left, operator, right) return left

And last we need to slightly change parse_expression() introducing a check on the literal token value. This happens because I decided to identify everything with a literal, so the method has to rule out every literal it is not interested to manage. If you introduce specific tokens for operations, parenthesis, etc., this change is not required (but you won't use clex.LITERAL at that point).

def parse_expression(self): left = self.parse_term() next_token = self.lexer.peek_token() if next_token.type == clex.LITERAL and next_token.value in ['+', '-']: operator = self.parse_addsymbol() right = self.parse_expression() return BinaryNode(left, operator, right) return left Level 11 - Priorities

Another feature that comes for free with the previous changes, as the first thing that parse_expression() does is to run parse_term(), and the first thing the latter does is to run parse_factor(), which in turn manages expressions between parenthesis. If the expression is enclosed between parenthesis the parse_factor() method doesn't call parse_expression() and just returns the integer.

Level 12 - Unary operators

The minus unary operator uses a literal that we already manage in the lexer, so there is nothing to do there. The first test I gave you checks if the parser can process a factor in the form -5.

The current implementation of parse_factor() processes either an expression enclosed between parenthesis or an integer, and actually the test doesn't pass, complaining against the minus sign not being a valid integer with base 10. The solution is pretty straightforward, as it is enough to add another if that manages the minus sign. When we encounter such a sign, however, wwe have to return a different type of node, as the test states, so we also have to introduce the relative class.

class UnaryNode(Node): node_type = 'unary' def __init__(self, operator, content): self.operator = operator self.content = content def asdict(self): result = { 'type': self.node_type, 'operator': self.operator.asdict(), 'content': self.content.asdict() } return result class CalcParser: def __init__(self): self.lexer = clex.CalcLexer() def parse_addsymbol(self): t = self.lexer.get_token() return LiteralNode(t.value) def parse_integer(self): t = self.lexer.get_token() return IntegerNode(t.value) def parse_factor(self): next_token = self.lexer.peek_token() if next_token.type == clex.LITERAL and next_token.value == '-': operator = self.parse_addsymbol() factor = self.parse_factor() return UnaryNode(operator, factor) if next_token.type == clex.LITERAL and next_token.value == '(': self.lexer.get_token() expression = self.parse_expression() self.lexer.get_token() return expression return self.parse_integer()

The second test passes automatically because parse_factor() intercepts the - literal before the ( one.

The visitor, then, has to be updated with the new type of unary node. The new visitor is then

class CalcVisitor: def visit(self, node): if node['type'] == 'integer': return node['value'], node['type'] if node['type'] == 'unary': operator = node['operator']['value'] cvalue, ctype = self.visit(node['content']) if operator == '-': return - cvalue, ctype if node['type'] == 'binary': lvalue, ltype = self.visit(node['left']) rvalue, rtype = self.visit(node['right']) operator = node['operator']['value'] if operator == '+': return lvalue + rvalue, ltype elif operator == '-': return lvalue - rvalue, ltype elif operator == '*': return lvalue * rvalue, ltype elif operator == '/': return lvalue // rvalue, ltype

Now the unary plus is easy to sort out, as we just need to take it into account in parse_factor() along with the unary minus.

def parse_factor(self): next_token = self.lexer.peek_token() if next_token.type == clex.LITERAL and next_token.value in ['-', '+']: operator = self.parse_addsymbol() factor = self.parse_factor() return UnaryNode(operator, factor) if next_token.type == clex.LITERAL and next_token.value == '(': self.lexer.get_token() expression = self.parse_expression() self.lexer.get_token() return expression return self.parse_integer()

And the visitor is missing a single return after the if statement that deals with the unary minus.

if node['type'] == 'unary': operator = node['operator']['value'] cvalue, ctype = self.visit(node['content']) if operator == '-': return - cvalue, ctype return cvalue, ctype Final words

Well, we have a pretty decent calculator now, don't we? Stay tuned, as in the next instalment we will explore variables and postfix operators. Get in touch if you want to discuss your solution or if you have questions about the code I posted here.


Feel free to use the blog Google+ page to comment the post. Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

Categories: FLOSS Project Planets

Jonathan Dowland: Electric Dreams

Planet Debian - Tue, 2017-10-17 07:33

No spoilers, for those who have yet to watch it...

Channel 4 have been broadcasting a new 10-part series called Electric Dreams, based on some of the short fiction of Philip K Dick. The series was commissioned after Channel 4 lost Black Mirror to Netflix, perhaps to try and find something tonally similar. Electric Dreams is executive-produced by Brian Cranston, who also stars in one of the episodes yet to broadcast.

I've read all of PKD's short fiction1 but it was a long time ago so I have mostly forgotten the stories upon which the series is based. I've quite enjoyed going back and re-reading them after watching the corresponding episodes to see what changes they've made. In some cases the changes are subtle or complementary, in other cases they've whittled the original story right out and installed a new one inside the shell. A companion compilation has been published with just the relevant short stories in it, and from what I've seen browsing it in a book shop it also contains short introductions which might be worth a read.

Things started strong with The Hood Maker, which my wife also enjoyed, although she was disappointed to realise we wouldn't be revisiting those characters in the future. The world-building was strong enough that it seemed like a waste for a single episode.

My favourite episode of those broadcast so far was The Commuter, starring Timothy Spall. The changes made were complementary and immensely expanded the emotional range of the story. In some ways, a key aspect of the original story was completely inverted, which I found quite funny: my original take on Dick's story was Dick implying a particular outcome was horrific, whereas it becomes desirable in the TV episode.

Episode 4, Crazy Diamond

One of the stories most hollowed-out was Sales Pitch which was the basis for Tony Grisoni’s episode Crazy Diamond, starring Steve Buscemi and Sidse Babett Knudsen. Buscemi was good but Knudsen totally stole every frame she was in. Fans of the cancelled Channel 4 show Utopia should enjoy this one: both were directed by Marc Munden and the directing, photography and colour balance really recall it.

The last episode broadcast was Real Life directed by Ronald D Moore of Battlestar Galactica reboot fame and starring Anna Paquin. Like Sales Pitch it bears very little resemblance to the original story. It played around with similar ideas explored in a lot of Sci-Fi movies and TV shows but left me a little flat; I didn't think it contributed much that I hadn't seen before. I was disappointed that there was a relatively conclusive ending. There was a subversive humour in the Dick short that was completely lost in the retelling. The world design seemed pretty generic.

I'm looking forward to Autofac, which is one of the shorts I can remember particularly enjoying.

  1. as collected in the 5 volumes of The Collected Stories of Philip K Dick, although I don't doubt there are some stragglers that were missed out when that series was compiled. ↩

Categories: FLOSS Project Planets

Ruwan Linton: Digital Transformation through Composable Integration

Planet Apache - Tue, 2017-10-17 06:38
Digitalization is heavily used in enterprises today to achieve business success. Business entities which do not embrace this change are losing their market share and going down day-by-day, as the human society is now experiencing digitalization at a global scale. This experience starts with all day-to-day activities to the major political, industrial, informational, educational and even cultural engagements. In essence, we are experiencing a Digital Revolution.

We are experiencing a Digital RevolutionThere are many examples of businesses not being able to transform itself with the emerging technologies and be defeated by their competitors:
  • Kodak defeated by the Digital Camera, mainly by Nikon and Canon
  • Nokia defeated by the smartphones, mainly by Samsung and Apple
  • BlockBuster Video defeated by online streaming, mainly by Netflix
  • Borders Books and Music defeated by online Bookstores, mainly by Amazon
And this list continues..
Digital TransformationWith digitalization, consumer expectations on the quality of a service and the speed or the turnover time of the service have increased dramatically. Lack of adherence to this change is usually seen by the consumer market as lack of innovation in products and services. The process of shifting from the existing business methodologies to the new digitalized business methodologies or offerings is considered as Digital Transformation. A more formalized definition to this could be given as;
“Digital Transformation is the use of technology to radically improve performance or reach of the enterprise”Let me take an example from the FIT (Fully Independent Traveller) domain. Assume that there is a good hotel booking service, available probably over the internet and many other media (such as via a call-center, a physical office, etc.) and they have had a very good market share of the hotel bookings. However if they continue to provide the same service that they have used to be offering, they will sure be losing their market share as there are competing services emerging to provide better QoS, lower response times and possibly novel and more convenient media (such as mobile applications); and most importantly, a better experience for the bookings through the use of new technology.
Success of the business relies on the satisfaction of the consumers

So how could our hotel booking service leverage digitalization to achieve this innovation for the business success of their products and services? Usually, this is the responsibility of the CIO and CTO, who should look at the following 3 key areas in terms of transforming the existing business into a Digital Business.

Customer ExperienceDigital advances such as analytics, social media, mobile applications and other embedded devices help enterprises to improve customer experience.Operational ProcessCIOs are utilizing technology to be competitive in the market, and the urge to improve business performance also leads executives to think of improving internal processes with digitalization.Business ModelBusiness processes/models and marketing models have to be adapted to support the new information age and transform themselves to seamlessly function with the rest of the digitalization stream.However digital transformation is not just the use of technology to achieve these qualities; rather, the complete business has to operate as one single system, from an external consumer’s perspective. That requires existing systems, new components, mobile applications and the bleeding edge IoT devices to be able to intercommunicate.System IntegrationWith the rise of Information Technology, large businesses started to benefit by introducing software systems to accomplish different tasks within the enterprise. They have introduced different monolithic systems such as HR management systems, SAP, ERP systems, CRM systems and many more. However these systems were designed to perform a specific tasks, and later on, to achieve the intercommunication of these incompatible systems, different concepts had to be introduced. Starting with manual data entry between different systems, these gradually evolved into EAI, and have been driven towards API Management and Microservices, with ETL, EI and ESB in the middle. The formal definition of system integration could be presented as;
“System Integration is the process of linking together different computing systems and software applications functionally to act as a coordinated whole”Integration in the enterprise domain has now been further expanded to mobile devices as well as IoT through APIs and Microservices, in order to meet consumer experience expectations.
So in essence, integration of systems, APIs and devices plays a vital role in Digital Transformation as it essentially requires all these to connect with each other and intercommunicate. Ability to seamlessly integrate existing systems without much overhead is the key to a successful implementation of Digital Transformation.The other aspect of this equilibrium function of business success is the Time to Market factor, which requires integration and all these new technology usages to be adopted as fast as possible. However, these integration problems require some careful design, together with a good amount of development to achieve protocol and transport bridging, data transformation, routing and other required integration functionalities, despite the availability of dozens of frameworks and products to facilitate the same. 
Composable IntegrationIn order to reduce this integration development time, AdroitLogic has developed a lean framework for integration named Project-X, with a rich ecosystem of tooling around it. Project-X facilitates 3 building blocks for integration:
Connectors & Processors are used to compose integration flows, without having to write a single line of code!ConnectorsConnectors could be used either to accept messages/events from outside (via Ingress Connectors) or to send out/invoke external services/functionalities (via Egress Connectors). In the rare case of not being able to find the ideal match in the existing connector palette, you could easily implement your own reusable connector.ProcessorsAny processing of an accepted message such as conditional evaluations, routing, transformations, value extractions, composition, data enrichment, validation etc., could be achieved by the army of pre-built processors. In the rare case of not being able to find the most suitable processor to implement your functionality, you could implement your own reusable processor to be included in your integration flow.FeaturesA set of utility functions available to be utilized by the processors and connectors, all of which could also be utilized by any custom connector or processor that you want to write. On top of that, you could also write your own features, which might utilize existing features in turn.All these pieces are seamlessly assembled via a composable drag-n-drop style integration flow editor named UltraStudio, based on the IDE champion IntelliJ IDEA, allowing you to compose your integration flows in minutes, test and debug it on the IDE itself, and build the final artifact using Maven to deploy it into the UltraESB-X lean runtime container for production execution.
Compose your integration flow, test and debug it on the IDE itself prior to deploymentYou can pick and choose the relevant connectors and processors to be utilized in your project, from the existing connector and processor store, and the project artifact will be built as a self-contained bundle which could be deployed in the integration runtime without any hassle of adding other required drivers or 3rd party jar files; yet the runtime will have that — and only that — set of dependencies, making your execution runtime as lean as possible. Further, this makes the project lifecycle and maintainability of the solution more robust, as the project could use your existing version control systems and continuous integration management to benefit from the collaboration techniques that you have already been practicing.AdroitLogic is in the process of building the 4th layer on top of this, named Templates. These templates will have reusable, parameterized patterns (or frameworks) for building your integration flows. For example, a solution which requires guaranteed delivery to a defined set of downstream systems with specific mapping and filtering criteria, together with validation from a given upstream system with traceability and statistics, could utilize an existing template and just compose the mapping and filtering criteria to implement the whole solution in a matter of minutes.In conclusion, if your organization has not yet started on the digital transformation, this is high time that you consider stepping up your pace. While this will have multiple streams of transformation and a lot of impact on the way the business currently operates, one good initiating point is to start integrating your existing systems to work seamlessly, and facilitating the ability to connect with your partners and consumers through the latest technologies to improve consumer experience.
Categories: FLOSS Project Planets

Appnovation Technologies: Appnovator Spotlight: Paulo Gomes

Planet Drupal - Tue, 2017-10-17 06:35
Appnovator Spotlight: Paulo Gomes Who are you? What's your story? My name is Paulo Gomes, I am from Portugal and moved to the UK with my wife in 2016 to join Appnovation crew. I am an tech and web enthusiast since the 90's (so not too old and not too young), I graduated in Computers and Management in 2002, after that worked in many places and companies, as freelancer, trainer and t...
Categories: FLOSS Project Planets

Python Software Foundation: Why Become A PyCon Sponsor?

Planet Python - Tue, 2017-10-17 06:15

Sponsors help keep PyCon affordable and accessible to the widest possible audience. Sponsors are what make this conference possible. From low ticket prices to financial aid, to video recording, the organizations who step forward to support PyCon, in turn, support the entire Python community. They make it possible for so many to attend, for so many to be presenters, and for the people at home to watch along.

The benefits of sponsorship are many - here are a few to consider:
  • Being part of the biggest and most prestigious Python conference in the world.

  • Being matched with those who could potentially become clients.

  • Staying in front of your current customers - 3300+ attendees will see your products. 
Building relationships with the Python community - people look to sponsors to see who is using and supporting Python.

  • Recruiting - if you’re hiring, PyCon is the place to be.

  • A private meeting or interview room to conduct business onsite. 

  • The opportunity for innovators and practitioners in your company to talk about how you’re using Python.

Depending on your level of sponsorship, packages include complimentary conference passes, booth space, lead retrieval scanners, speaking opportunities, and a table in the Job Fair.

We’re flexible and willing to work with you to design the sponsorship package to fulfill your business needs. Starting a discussion now is a great way to design a more custom program for you. Our sponsorship prospectus can be found here: https://us.pycon.org/2018/sponsors/prospectus/

PyCon 2018
Huntington Convention Center May 9th - May 17th, 2018 Cleveland, Ohio USA
For more information please contact: pycon-sponsors@python.org

                                                                  Photo by Mike Pirnat
Categories: FLOSS Project Planets

GUIX Project news: Coming events

GNU Planet! - Tue, 2017-10-17 06:00

Guix will be present on a few venues in the coming weeks:

  1. On October 23rd, I (Ludovic Courtès) will be at GPCE, an academic conference co-located with SPLASH in Vancouver, Canada. I will present the paper Code Staging in GNU Guix, which discusses the motivation for and genesis of G-expressions, as well as recent improvements. It’s an honor to be presenting before an audience of experts in the field!
  2. Christopher Baines will be at freenode #live in Bristol, UK, among well-known free software activists from a variety of organizations and projects. Christopher will give a talk on October 29th to give an overview of Guix and GuixSD.
  3. On October 31st, Ricardo Wurmus, Jan Nieuwenhuizen, and possibly more Guix hackers will join a dozen free software projects at the third Reproducible Build Summit in Berlin, Germany. As in previous years, we expect it to be a good time to share tips & tricks as well as a longer-term vision with our fellow hackers!

If you’re around in Vancouver, Bristol, or Berlin, let’s get in touch! :-)

About GNU Guix

GNU Guix is a transactional package manager for the GNU system. The Guix System Distribution or GuixSD is an advanced distribution of the GNU system that relies on GNU Guix and respects the user's freedom.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. Guix uses low-level mechanisms from the Nix package manager, except that packages are defined as native Guile modules, using extensions to the Scheme language. GuixSD offers a declarative approach to operating system configuration management, and is highly customizable and hackable.

GuixSD can be used on an i686 or x86_64 machine. It is also possible to use Guix on top of an already installed GNU/Linux system, including on mips64el, armv7, and aarch64.

Categories: FLOSS Project Planets

Dropsolid: Load testing Drupal with Blazemeter and JMeter

Planet Drupal - Tue, 2017-10-17 05:59
17 Oct Load testing Drupal with Blazemeter and JMeter Niels A Tech

When going live with a big project, it is all about reassuring the client that the project will be able to handle all those excited visitors. To achieve that state of zen, it is paramount that you do a load test. The benefits of load tests go beyond peace of mind, however. For example, it enables you to spot issues that only happen during high load or let’s you spot bottlenecks in the infrastructure setup. The added bonus is that you can bask in the glory of your high-performance code - on the condition the test doesn’t fail, of course.

Need help with your load and performance testing?
Contact us 


When doing a load test it is important to do the following steps:

  • Analyse existing data
  • Prepare tests
  • Set up tools
  • Run the tests
  • Analyse the results


Analyse existing data

If you are in luck, you will already  have historic data available to use from Google Analytics. If this isn’t the case, you’ll have to get in touch with your client and ask a few to-the-point questions to help you estimate all the important metrics that I’ll be covering in this post.


A couple of tips I can give if you lack the historic data:

  • Ask if the client has a mailing (digital or old-school) list and how many people are on it
  • If you have made comparable sites in the past, look at their Google Analytics data
  • Ask the client how they are going to announce their new website
  • When you are working on an estimate, it is always better to add an extra 15% to it. Better safe than sorry!

The first thing you need to do, is set a reference frame. Pick a date range that has low activity as well as the highest activity you can find. Then start putting that data into a spreadsheet, as pictured below:

You can download an example copy of the file here


The most important metrics we are going to calculate are:

  • Peak concurrent user (Hourly sessions x Average sessions Duration / 3600)
  • Peak page views per second


The values you need to find or estimate are:

  • Peak daily page views
  • Peak hourly page views
  • Total page view for period
  • Peak hourly sessions
  • Total amount of sessions
  • Average session duration in seconds

As you can see, we mainly focus on the peak activity, because you test with the worst-case scenario in mind - which is, funnily enough, usually the best-case scenario for your client.

Before we start preparing our test, it is also handy to check which pages receive the most traffic. This benefits the validity of your test scenario.


Prepare the tests

For our tests we are going to start out with Apache JMeter, which you can grab here.

With JMeter you can test many different applications/server/protocol types, but we’re going to use it to make a whole lot of HTTP requests.

Make sure you have the required Java library and go boot up the ApacheJMeter.jar file.


Adding and configuring a Thread Group

Start by adding a Thread Group to your test plan by right clicking your Test plan and selecting Add > Threads (Users) > Thread Group


Eventually you will need to fill in the number of (concurrent) users and ramp-up period based on your analysis, but for now keep it low for debugging your test.


Adding and configuring User-Defined Variables

Then right click the thread group to add User Defined Variables (Add > Config Element > User Defined Variables).

Add two variables named url and protocol and assign them a value.

Using these user-defined variables makes it easy to choose another environment to test on. It avoids the painstaking and error-prone work of finding all references and changing them manually.

You can use these variables in input fields in your test by doing this: ${url} or ${protocol}


Adding and configuring HTTP config elements

 Next up, you need to add the following HTTP config elements to your thread group:

  • HTTP Request Defaults
  • HTTP Header Manager
  • HTTP Cookie Manager

On the first one, you use your variables to fill in the protocol and the server name.

On the second one, you can set default headers for each one of your requests. See the screenshot below for what I’ve put in default.

For the third one, you only select cookie policy: standard.


A simple page request sampler

Right-click your test again and add the HTTP request sampler (Add > Sampler > HTTP Request).

Here we are going to call the home page. The only things you need to set here are:

  • Method: GET
  • Path: /

We don’t fill in the protocol or server name because this is already covered by our HTTP Request Defaults.


Posting the contact form

In this one we are going to submit the contact form (which is located at www.example.com/contact), so add another HTTP Request like we did before. Now only fill in the following values:

  • Method: POST
  • Path: /contact
  • Follow redirects: True
  • Use KeepAlive: True

In order for Drupal to accept the submit, we need to add some parameters to our post, like this:

The important ones here are form_build_id and form_id. You can manually get the form id because it always stays the same. The form build ID can vary, so we need to extract this from the page. We’ll do this using the CSS/JQuery Extractor (right-click your HTTP Request sampler: Add > Post Processors > CSS/JQuery Extractor)

Configure it like the screenshot below:

It will now get that form_build_id from the page and put into a variable the sampler can use.$


Posting some Ajax on the form

Imagine our contact form has some Ajax functionality and we also want to test this. The way we go about it is identical to posting the regular form like we did before. The only difference is the post parameters, the path and an extra HTTP Header Manager.

You should set the path in your sampler to: /system/ajax

Then right click your sampler to add your new HTTP Header Manager (Add > Config Element > HTTP Header Manager). Configure it like shown in the screenshot:

  Saving the results of your test

Now that we’ve configured samplers, we need to add some listeners. You can add these listeners everywhere, but in our example we’ve added it to the test in a whole.


We’ll add three listeners:

  • View Results in Table:
    • Show every request in a table format
    • Handy for getting some metrics like latency and connect time
  • Simple Data Writer:
    • Writes test data to a file
    • Handy for debugging when using Blazemeter (check out this link)
    • Just load the file into the View Results Tree
  • View Results Tree:
    • It shows you the actual response and request.
    • Uses a lot of resources (so only good for debugging)


There is a lot more you can do with JMeter. You can read all about it here.

Test-run the test

Now that we’ve configured our test it is time to try it out. So make sure not to put too much concurrent users in there. Just run the test by pressing the green ‘Play’ icon.

If you get errors, debug them using the feedback you got from your listeners.

As this wise man once said: "Improvise. Adapt. Overcome."

After you’ve validated your test, it’s always handy to turn up the concurrent users until your local site breaks. It’ll give you a quick idea of where a possible bottleneck could be.

Just a small warning: doing that load test on your local machine (running the test and the webserver) will take up a lot of resources and can give you skewed results.

You can download an example here.


Set up tools Load testing with Blazemeter

When you have a project that will have a lot of concurrent users, your computer is most likely not able to handle doing all those calls and that is why it is good to test from a distributed setup like Blazemeter does.


You can have multiple computers running the same test with only a part of the concurrent users or you can pay for a service like Blazemeter.


The downside of using multiple computers is that they still use the same corporate WiFi or ethernet, blocking yourself possibly to the lowest common denominator, which is most likely unknown and could cause trouble that might skew your test. On top of that you will also have to aggregate all those results yourself, costing you precious time.


For us the mayor benefits of Blazemeter are the following:

  • Simulate a massive amount of concurrent users with little hassle
  • Persistence of test results and comparison between tests
  • Executive report to deliver to a technical savvy client
  • Sandbox mode tests that don’t count against your monthly testing quota


Adding your JMeter test in Blazemeter is very easy and straightforward. Just click ‘Create Test’ in the menu and select JMeter Test.

Upload the file and you can start to configure your test to reflect your test scenario from the analysis chapter. We suggest to choose to ‘Originate a load’ from a service that is closest to your target population.

Before you run your test, it is important to have set up your monitoring of the environment you want to test.


Monitoring performance

At Dropsolid, we like to use New Relic to monitor performance of our environments but you could also use open source tools like Munin.

The most important factors in your decision of monitoring tool should be:

  • Persistence of monitoring data
  • Detail of monitoring data
  • Ease of use

If you are using New Relic, we recommend to install both APM and Server. The added value of having APM is that you can quickly get an overview of possible bottlenecks in PHP and MySQL.


Run the test

Now that everything is set up, it is important to have an environment that is a perfect copy of your production environment. That way you can easily optimize your environment without having to wait for a good moment to restart your server.

Run your test, sit back and relax.


Analyse the results

If everything has gone according to plan, you should now have reports from both Blazemeter and New Relic.

Blazemeter report of a test of 854 concurrent usersNew Relic monitoring during the same test

If your server was able to handle the peak amount of users, then your job is done and you can inform the client that they can rest assured that it won’t go down.

If your server couldn’t handle it, it is time to compare the results from Blazemeter and New Relic to find out where your bottleneck is.

Common issues are the following:

  • Not the right memory allocation between parts of the stack.
  • Misconfiguration of your stack. For example, MySQL has multiple example configuration files for different scenarios
  • Not using extra performance enhancing services like varnish, memcache, redis,...
  • Horrible code

If the issue is horrible code, then use tools like xhprof or blackfire.io to profile your code.

Need expert help with your performance tests? Just get in touch!

Contact us for performance testing 

Final note

As Colin Powell once said: "There are no secrets to success. It is the result of preparation, hard work and learning from failure." That is exactly what we did here: we prepared our test thoroughly, we tested our script multiple times and adapted when it failed.

Categories: FLOSS Project Planets

S. Lott: Why I like Functional Composition

Planet Python - Tue, 2017-10-17 04:00
After spending years developing a level of mastery over Object Oriented Design Patterns, I'm having a lot of fun understanding Functional Design Patterns.

The OO Design Patterns are helpful because they're concrete expressions of the S. O. L. I. D. design principles. Much of the "Gang of Four" book demonstrates the Interface Segregation, Dependency Injection, and Liskov Substitution Principles nicely. They point the way for implementing the Open/Closed and the Single Responsibility Principles.

For Functional Programming, there are some equivalent ideas, with distinct implementation techniques. The basic I, D, L, and S principles apply, but have a different look in a functional programming context. The Open/Closed principle takes on a radically different look, because it turns into an exercise in Functional Composition.

I'm building an Arduino device that collects GPS data. (The context for this device is the subject of many posts coming in the future.)

GPS devices generally follow the NMEA 0183 protocol, and transmit their data as sentences with various kinds of formats. In particular, the GPRMC and GPVTG sentences contain speed over ground   (SOG) data.

I've been collecting data in my apartment. And it's odd-looking. I've also collected data on my boat, and it doesn't seem to look quite so odd. Here's the analysis I used to make a more concrete conclusion.

def sog_study(source_path = Path("gps_data_gsa.csv")):
with source_path.open() as source_file:
rdr = csv.DictReader(source_file)
sog_seq = list(map(float, filter(None, (row['SOG'] for row in rdr))))
print("max {}\tMean {}\tStdev {}".format(
max(sog_seq), statistics.mean(sog_seq), statistics.stdev(sog_seq)))

This is a small example of functional composition to build a sequence of SOG reports for analysis.

This code opens a CSV file with data extracted from the Arduino. There was some reformatting and normalizing done in a separate process: this resulted in a file in a format suitable for the processing shown above.

The compositional part of this is the list(map(float, filter(None, generator))) processing.

The (row['SOG'] for row in rdr) generator can iterate over all values from the SOG column. The filter(None, generator) will drop all None objects from the results, assuring that irrelevant sentences are ignored.

Given an iterable that can produce SOG values, the map(float, iterable) will convert the input strings into useful numbers. The surrounding list() creates a concrete list object to support summary statistics computations.

I'm really delighted with this kind of short, focused functional programming.

"But wait," you say. "How is that anything like the SOLID OO design?"

Remember to drop the OO notions. This is functional composition, not object composition.

ISP: The built-in functions all have well-segregated interfaces. Each one does a small, isolated job.

LSP: The concept of an iterable supports the Liskov Substitution Principle: it's easy to insert additional or different processing as long as we define functions that accept iterables as an argument and yield their values or return an iterable result.

For example.

def sog_gen(csv_reader):
for row in csv_reader:
yield row['SOG']

We've expanded the generator expression, (row['SOG'] for row in rdr), into a function. We can now use sog_gen(rdr) instead of the generator expression. The interfaces are the same, and the two expressions enjoy Liskov Substitution.

To be really precise, annotation with type hints can clarify this.  Something like sog_gen(rdr: Iterable[Dict[str, str]]) -> Iterable[str] would clarify this.

DIP: If we want to break this down into separate assignment statements, we can see how a different function can easily be injected into the processing pipeline. We could define a higher-order function that accepted functions like sog_gen, float, statistics.mean, etc., and then created the composite expression.

OCP: Each of the component functions is closed to modification but open to extension. We might want to do something like this: map_float = lambda source: map(float, source). The map_float() function extends map() to include a float operation. We might even want to write something like this.  map_float = lambda xform, source: map(xform, map(float, source)). This would look more like map(), with a float operation provided automatically.

SRP: Each of the built-in functions does one thing. The overall composition builds a complex operation from simple pieces.

The composite operation has two features which are likely to change: the column name and the transformation function. Perhaps we might rename the column from 'SOG' to 'sog'; perhaps we might use decimal() instead of float(). There are a number of less-likely changes. There might be a more complex filter rule, or perhaps a more complex transformation before computing the statistical summary.  These changes would lead to a different composition of the similar underlying pieces.
Categories: FLOSS Project Planets

Russ Allbery: Bundle haul

Planet Debian - Tue, 2017-10-17 01:38

Confession time: I started making these posts (eons ago) because a close friend did as well, and I enjoyed reading them. But the main reason why I continue is because the primary way I have to keep track of the books I've bought and avoid duplicates is, well, grep on these posts.

I should come up with a non-bullshit way of doing this, but time to do more elegant things is in short supply, and, well, it's my blog. So I'm boring all of you who read this in various places with my internal bookkeeping. I do try to at least add a bit of commentary.

This one will be more tedious than most since it includes five separate Humble Bundles, which increases the volume a lot. (I just realized I'd forgotten to record those purchases from the past several months.)

First, the individual books I bought directly:

Ilona Andrews — Sweep in Peace (sff)
Ilona Andrews — One Fell Sweep (sff)
Steven Brust — Vallista (sff)
Nicky Drayden — The Prey of Gods (sff)
Meg Elison — The Book of the Unnamed Midwife (sff)
Pat Green — Night Moves (nonfiction)
Ann Leckie — Provenance (sff)
Seanan McGuire — Once Broken Faith (sff)
Seanan McGuire — The Brightest Fell (sff)
K. Arsenault Rivera — The Tiger's Daughter (sff)
Matthew Walker — Why We Sleep (nonfiction)

Some new books by favorite authors, a few new releases I heard good things about, and two (Night Moves and Why We Sleep) from references in on-line articles that impressed me.

The books from security bundles (this is mostly work reading, assuming I'll get to any of it), including a blockchain bundle:

Wil Allsop — Unauthorised Access (nonfiction)
Ross Anderson — Security Engineering (nonfiction)
Chris Anley, et al. — The Shellcoder's Handbook (nonfiction)
Conrad Barsky & Chris Wilmer — Bitcoin for the Befuddled (nonfiction)
Imran Bashir — Mastering Blockchain (nonfiction)
Richard Bejtlich — The Practice of Network Security (nonfiction)
Kariappa Bheemaiah — The Blockchain Alternative (nonfiction)
Violet Blue — Smart Girl's Guide to Privacy (nonfiction)
Richard Caetano — Learning Bitcoin (nonfiction)
Nick Cano — Game Hacking (nonfiction)
Bruce Dang, et al. — Practical Reverse Engineering (nonfiction)
Chris Dannen — Introducing Ethereum and Solidity (nonfiction)
Daniel Drescher — Blockchain Basics (nonfiction)
Chris Eagle — The IDA Pro Book, 2nd Edition (nonfiction)
Nikolay Elenkov — Android Security Internals (nonfiction)
Jon Erickson — Hacking, 2nd Edition (nonfiction)
Pedro Franco — Understanding Bitcoin (nonfiction)
Christopher Hadnagy — Social Engineering (nonfiction)
Peter N.M. Hansteen — The Book of PF (nonfiction)
Brian Kelly — The Bitcoin Big Bang (nonfiction)
David Kennedy, et al. — Metasploit (nonfiction)
Manul Laphroaig (ed.) — PoC || GTFO (nonfiction)
Michael Hale Ligh, et al. — The Art of Memory Forensics (nonfiction)
Michael Hale Ligh, et al. — Malware Analyst's Cookbook (nonfiction)
Michael W. Lucas — Absolute OpenBSD, 2nd Edition (nonfiction)
Bruce Nikkel — Practical Forensic Imaging (nonfiction)
Sean-Philip Oriyano — CEHv9 (nonfiction)
Kevin D. Mitnick — The Art of Deception (nonfiction)
Narayan Prusty — Building Blockchain Projects (nonfiction)
Prypto — Bitcoin for Dummies (nonfiction)
Chris Sanders — Practical Packet Analysis, 3rd Edition (nonfiction)
Bruce Schneier — Applied Cryptography (nonfiction)
Adam Shostack — Threat Modeling (nonfiction)
Craig Smith — The Car Hacker's Handbook (nonfiction)
Dafydd Stuttard & Marcus Pinto — The Web Application Hacker's Handbook (nonfiction)
Albert Szmigielski — Bitcoin Essentials (nonfiction)
David Thiel — iOS Application Security (nonfiction)
Georgia Weidman — Penetration Testing (nonfiction)

Finally, the two SF bundles:

Buzz Aldrin & John Barnes — Encounter with Tiber (sff)
Poul Anderson — Orion Shall Rise (sff)
Greg Bear — The Forge of God (sff)
Octavia E. Butler — Dawn (sff)
William C. Dietz — Steelheart (sff)
J.L. Doty — A Choice of Treasons (sff)
Harlan Ellison — The City on the Edge of Forever (sff)
Toh Enjoe — Self-Reference ENGINE (sff)
David Feintuch — Midshipman's Hope (sff)
Alan Dean Foster — Icerigger (sff)
Alan Dean Foster — Mission to Moulokin (sff)
Alan Dean Foster — The Deluge Drivers (sff)
Taiyo Fujii — Orbital Cloud (sff)
Hideo Furukawa — Belka, Why Don't You Bark? (sff)
Haikasoru (ed.) — Saiensu Fikushon 2016 (sff anthology)
Joe Haldeman — All My Sins Remembered (sff)
Jyouji Hayashi — The Ouroboros Wave (sff)
Sergei Lukyanenko — The Genome (sff)
Chohei Kambayashi — Good Luck, Yukikaze (sff)
Chohei Kambayashi — Yukikaze (sff)
Sakyo Komatsu — Virus (sff)
Miyuki Miyabe — The Book of Heroes (sff)
Kazuki Sakuraba — Red Girls (sff)
Robert Silverberg — Across a Billion Years (sff)
Allen Steele — Orbital Decay (sff)
Bruce Sterling — Schismatrix Plus (sff)
Michael Swanwick — Vacuum Flowers (sff)
Yoshiki Tanaka — Legend of the Galactic Heroes, Volume 1: Dawn (sff)
Yoshiki Tanaka — Legend of the Galactic Heroes, Volume 2: Ambition (sff)
Yoshiki Tanaka — Legend of the Galactic Heroes, Volume 3: Endurance (sff)
Tow Ubukata — Mardock Scramble (sff)
Sayuri Ueda — The Cage of Zeus (sff)
Sean Williams & Shane Dix — Echoes of Earth (sff)
Hiroshi Yamamoto — MM9 (sff)
Timothy Zahn — Blackcollar (sff)

Phew. Okay, all caught up, and hopefully won't have to dump something like this again in the near future. Also, more books than I have any actual time to read, but what else is new.

Categories: FLOSS Project Planets

Norbert Preining: Japanese TeX User Meeting 2017

Planet Debian - Tue, 2017-10-17 01:22

Last saturday the Japanese TeX User Meeting took place in Fujisawa, Kanagawa. For those who have been at the TUG 2013 in Tokyo you will remember that the Japanese TeX community is quite big and vibrant. On Saturday about 50 users and developers gathered for a set of talks on a variety of topics.

The first talk was by Keiichiro Shikano (鹿野 桂一郎) on using Markup text to generate (La)TeX and HTML. He presented a variety of markup formats, including his own tool xml2tex.

The second talk was my Masamichi Hosoda (細田 真道) on reducing the size of PDF files using PDFmark extraction. As a contributor to many projects including Texinfo and LilyPond, Masamichi Hosoda tells us horror stories about multiple font embedding in the manual of LilyPond, the permanent need for adaption to newer Ghostscript versions, and the very recent development in Ghostscript prohibiting the merge of font definitions in PDF files.

Next up was Yusuke Terada (寺田 侑祐) on grading exams using TeX. Working through hundreds and hundreds of exams and do the grading is something many of us are used to and I think nobody really enjoys it. Yusuke Terada has combined various tools, including scans, pdf merging using pdfpages, to generate gradable PDF which were then checked on an iPad. On the way he did hit some limits in dvipdfmx on the number of images, but this was obviously only a small bump on the road. Now if that could be automatized as a nice application, it would be a big hit I guess!

The forth talk was by Satoshi Yamashita (山下 哲) on the preparation of slides using KETpic. KETpic is a long running project by Setsuo Takato (高遠節夫) for the generation of graphics, in particular using Cinderella. KETpic and KETcindy integrates with lots of algebraic and statistical programs (R, Maxima, SciLab, …) and has a long history of development. Currently there are activities to incorporate it into TeX Live.

The fifth talk was by Takuto Asakura (朝倉 卓人) on programming TeX using expl3, the main building block of the LaTeX3 project and already adopted by many TeX developers. Takuto Asakura came to fame on this years TUG/BachoTeX 2017 when he won the W. J. Martin Prize for his presentation Implementing bioinformatics algorithms in TeX. I think we can expect great new developments from Takuto!

The last talk was by myself on fmtutil and updmap, two of the main management programs in any TeX installation, presenting the changes introduced over the last year, including the most recent release of TeX Live. Details have been posted on my blog, and a lengthy article in TUGboat 38:2, 2017 is available on this topic, too.

After the conference about half of the participants joined a social dinner in a nearby Izakaya, followed by a after-dinner beer tasting at a local craft beer place. Thanks to Tatsuyoshi Hamada for the organization.

As usual, the Japanese TeX User Meetings are a great opportunity to discuss new features and make new friends. I am always grateful to be part of this very nice community! I am looking forward to the next year’s meeting.

Categories: FLOSS Project Planets

Mike Driscoll: How to Watermark Your Photos with Python

Planet Python - Tue, 2017-10-17 01:15

When you look up photos online, you will notice that some of them are watermarked. A watermark is usually some text or a logo overlaid on the photo that identifies who took the photo or who owns the rights to the photo. Some professionals recommend adding watermarks to your photos before sharing them on social media to prevent other people from using your photos as their own and profiting off your work. Of course, watermarking can be removed fairly easily, so this isn’t as useful as it used to be as a digital rights tool.

Anyway, the Pillow package provides the tools you need to add watermarks to your photos! The first thing you need to do is install Pillow if you haven’t already:

pip install pillow

Once that’s installed, we can continue!

Adding a Text Watermark

We will start by just adding some text to a photograph. Let’s use this one of a the Yequina lighthouse that I took in Oregon:

Now we will add some text to the image. In this case, let’s add my domain to it: www.mousevspython.com

from PIL import Image from PIL import ImageDraw from PIL import ImageFont     def watermark_text(input_image_path, output_image_path, text, pos): photo = Image.open(input_image_path)   # make the image editable drawing = ImageDraw.Draw(photo)   black = (3, 8, 12) font = ImageFont.truetype("Pillow/Tests/fonts/FreeMono.ttf", 40) drawing.text(pos, text, fill=black, font=font) photo.show() photo.save(output_image_path)     if __name__ == '__main__': img = 'lighthouse.jpg' watermark_text(img, 'lighthouse_watermarked.jpg', text='www.mousevspython.com', pos=(0, 0))

Here we import a few classes from PIL: Image, ImageDraw and ImageFont. Then we create a function, watermark_text, that takes four arguments: the input image page, the output image path, the text to add to the image, and the position coordinates to put the text in.

Next we open our image. Then we basically make a copy of the image by redrawing it using the ImageDraw.Draw method. This allows us to add text to it more easily. Then we create a font using ImageFont.truetype. The font is one that is included with the Pillow library. We set the font size to 40 points as well. Finally we show the result and save it to disk. Here is the result:

Now let’s move on to add an image instead of just text!

Watermark with an Image

Most professional photographers end up watermarking their photos with a logo. Sometimes that includes a stylized “photo” version of their website. Adding a photo to another photo in Pillow is also quite easy. Let’s find out how to do it!

In this example, we will use one of my logos for the watermark image.

from PIL import Image   def watermark_photo(input_image_path, output_image_path, watermark_image_path, position): base_image = Image.open(input_image_path) watermark = Image.open(watermark_image_path)   # add watermark to your image base_image.paste(watermark, position) base_image.show() base_image.save(output_image_path)   if __name__ == '__main__': img = 'lighthouse.jpg' watermark_with_photo(img, 'lighthouse_watermarked2.jpg', 'watermark.png', position=(0,0))

Here we create a function that is very similar to the function in the last section, but instead of passing in text, we pass in the watermark’s file path. In the function, we open up the image we wish to watermark as well as the watermark image itself. Then we take the image to be watermarked and call its paste() method while passing in what we want to paste (i.e. the watermark) and the position we want the item pasted. Finally we show the image and save it. Here’s the result:

Well that didn’t work out the way I had envisioned it. As you can see, there’s a black background obscuring a lot more of the photo than I intended because when we pasted it, we didn’t account for transparency.

Watermark an Image with Transparency

Let’s create a new function that can use transparency so we remove all the black from the watermark. I found a solution for this issue on StackOverflow, so I took it and modified it slightly for this example

from PIL import Image   def watermark_with_transparency(input_image_path, output_image_path, watermark_image_path, position): base_image = Image.open(input_image_path) watermark = Image.open(watermark_image_path) width, height = base_image.size   transparent = Image.new('RGBA', (width, height), (0,0,0,0)) transparent.paste(base_image, (0,0)) transparent.paste(watermark, position, mask=watermark) transparent.show() transparent.save(output_image_path)     if __name__ == '__main__': img = 'lighthouse.jpg' watermark_with_transparency(img, 'lighthouse_watermarked3.jpg', 'watermark.png', position=(0,0))

In this code, we take in all the same arguments that we did in the previous example. This time we not only open up both images, but we also grab the width and height of the image that we want to watermark. Then we create a new image using the same width and height as the image we are watermarking. You will note that this image that we create is RGBA which means it has Red, Green and Blue with Alpha. Next we paste in the image we want to watermark starting at the upper left, which is (0,0). Then we paste in our watermark using the passed in position and we also mask the watermark with itself. Finally we show and save the image.

The resulting image looks like this:

Pretty cool, eh?

Wrapping Up

We covered 2 different methods of adding watermarks to your photographs in this article. In the first example, all we did was add the text of your choice to the image. The second example demonstrated a way to add an image watermark, but it didn’t account for alpha (transparency). We corrected that issue with the third example. I hope you found these examples helpful. Happy coding!

Related Reading

Categories: FLOSS Project Planets

François Marier: Checking Your Passwords Against the Have I Been Pwned List

Planet Debian - Tue, 2017-10-17 01:10

Two months ago, Troy Hunt, the security professional behind Have I been pwned?, released an incredibly comprehensive password list in the hope that it would allow web developers to steer their users away from passwords that have been compromised in past breaches.

While the list released by HIBP is hashed, the plaintext passwords are out there and one should assume that password crackers have access to them. So if you use a password on that list, you can be fairly confident that it's very easy to guess or crack your password.

I wanted to check my active passwords against that list to check whether or not any of them are compromised and should be changed immediately. This meant that I needed to download the list and do these lookups locally since it's not a good idea to send your current passwords to this third-party service.

I put my tool up on Launchpad / PyPI and you are more than welcome to give it a go. Install Postgres and Psycopg2 and then follow the README instructions to setup your database.

Categories: FLOSS Project Planets

Brad Lucas: Coin Market Cap Update

Planet Python - Tue, 2017-10-17 00:00

So, it looks like the CoinMarketCap has changed their site a bit and it breaks the coinmarketcap.py script I wrote back in July. Many thanks to cool007zqw for pointing this out by adding an issue on the repo.

On investigation it looks like CoinMarketCap changed the url to the page which the script was developed for.

The new page is as follows:


This does show the fragility of building scripts to pull pages from web sites. I do see that CoinMarketCap has an API over at https://coinmarketcap.com/api/ so that may be the way forward to build something more robust.

Categories: FLOSS Project Planets

Django Weblog: Django 2.0 beta 1 released

Planet Python - Mon, 2017-10-16 22:09

Django 2.0 beta 1 is an opportunity for you to try out the assortment of new features in Django 2.0.

Only bugs in new features and regressions from earlier versions of Django will be fixed between now and 2.0 final (also, translations will be updated following the "string freeze" when the release candidate is issued). The current release schedule calls for a release candidate in a month from now with the final release to follow about two weeks after that around December 1. Early and often testing from the community will help minimize the number of bugs in the release. Updates on the release schedule schedule are available on the django-developers mailing list.

As with all alpha and beta packages, this is not for production use. But if you'd like to take some of the new features for a spin, or to help find and fix bugs (which should be reported to the issue tracker), you can grab a copy of the beta package from our downloads page or on PyPI.

The PGP key ID used for this release is Tim Graham: 1E8ABDC773EDE252.

Categories: FLOSS Project Planets

Alex Grönholm: Automating packaging and uploading of binary wheels

Planet Python - Mon, 2017-10-16 21:37
There has been plenty of frustration surrounding Python libraries which contain C extensions. Users are frustrated because pre-built wheels are not available for their platform, requiring them to install a compatible compiler. This has been particularly painful for Windows users. On the developer side, projects that have gone through the trouble of providing binary wheels often have a rather complicated process for building and uploading them, requiring several manual steps to get it done. On the other hand, projects that choose not to build binary wheels sometimes have droves of users nagging them to build those wheels (yes, I've been guilty of that). All in all, nobody's happy.

What makes this situation particularly challenging is that there is no single CI service available for F/OSS projects which could build wheels for all three major platforms (Linux, macOS and Windows). Travis supports Linux and macOS but not Windows. Then there's AppVeyor which only supports Windows. So in order to make everyone happy, you would need to combine the powers of both continuous integration services.
An additional challenge has been the lack of coordination tools for ensuring that a new release is uploaded to PyPI only if all the build jobs succeed. Naive configurations (yes, I've been guilty of that too) build the wheels and upload them to PyPI independently. This can lead to situations where a build job legitimately fails in a way that should've been a release blocker, but the jobs that succeeded have already uploaded their artifacts to PyPI. Now you have a botched release in your hands.

What if I told you that there is a way to set up your project so that all you have to do is add a git tag and push to Github, and the wheels and the source distribution would automatically get built and uploaded to PyPI if (and only if) all goes well?

Yes, folks. It can be done. You just need an adventurous mind. Take the red pill and find out how deep the rabbit hole goes.

How it worksAs I hinted earlier, the recipe I'm about to present combines three important factors:
  1. Use of Travis's "Build Stages" feature
  2. Use of AppVeyor via its ReST API
  3. Use of an external storage service (Amazon S3 is used here, but it could be something else)
The gist is this: The Travis build first runs the tests against all supported Python versions. After the tests have finished successfully, Travis starts building wheels for Linux and macOS. Meanwhile, an additional job is started which sends a request to AppVeyor's ReST API which tells it to start a build against the current git changeset. It will then poll the status of the build on regular intervals until it finishes one way or another. If it fails, the Travis build is failed. If it succeeds, the build artifacts are downloaded to the container running the Travis build.
When all the wheels have been built, their respective build jobs will upload them to the shared storage. Then the final job is started which pulls all the artifacts from this storage and uploads them to PyPI.

Setting it upTl;dr: Go see the example project and adapt its configuration to your needs.

You will need to have the following set up before proceeding:
  • A project on Github
  • A PyPI account
  • An AppVeyor account
  • An AWS account (if S3 is used)
The use of Amazon's S3 could be replaced with any other storage service. However, the free tier on AWS should get you enough disk space to satisfy the needs of most projects. You do need to have a valid credit card, however.
You will need at least these two configuration files present in the project's root directory:
  • .travis.yml: build configuration for Travis (this is the most important part)
  • appveyor.yml: build configuration for AppVeyor
You can copy the linked files to your own project as a base. Just remember to replace the environment variables in .travis.yml (or remove them altogether, as explained below).

Travis setupIf your project does not yet have Travis integration enabled, you need to do the following:
  1. Go to your project's settings on Github
  2. Click on "Integrations and services"
  3. Click on "Add service"
  4. Choose "Travis CI" and enter your Github password when prompted to do so
  5. Go to your Travis profile settings on their site
  6. Click on "Sync account" (at the top right) to refresh the list of projects
  7. Find your project on the list after the sync is complete and turn the switch on
  8. Click on the cogwheel next to your project's name to enter the settings page
The following Travis project settings are recommended:
Next, you will need to define the following environment variables:
  • APPVEYOR_SLUG (your project name on AppVeyor)
  • APPVEYOR_ACCOUNT (your account name on AppVeyor)
  • TWINE_USERNAME (your PyPI user name)
  • TWINE_PASSWORD (your PyPI password)
  • AWS_ACCESS_KEY_ID (the access key ID from AWS, for shared storage)
  • AWS_SECRET_ACCESS_KEY (the secret key from AWS, for shared storage)
There are two ways you can provide your build jobs environment variables:
  1. Add them to your .travis.yml file, encrypting any confidential ones like passwords
  2. Add them on your Travis project settings page
To encrypt an environment variable, you will need to have Travis's command line client installed. Then, you can do something like this:echo -n TWINE_PASSWORD=foobarbaz | travis encryptThis will output an encrypted secret which you can paste into .travis.yml. Note the importance of the -n switch, as without that a newline character would be added to the end which would cause the wrong text to be encrypted.

AppVeyor setupAssuming you have your AppVeyor account set up, you need to add your project to it. First, go to the Projects section in the top level menu and click "New Project". Then select Github and pick the project from that list.

Next, you need to disable the web hook AppVeyor just added to your Github project. This is necessary because the AppVeyor build should only be triggered by the wheel build stage on Travis. On Github, go to Settings -> Webhooks and edit the AppVeyor hook. Uncheck the "Active" check box and press "Update webhook", as shown below:
That's it for the AppVeyor configuration, unless your project has some special requirements.

AfterthoughtsI hope this will lower the barrier for projects to start producing binary wheels.
It should be noted that some projects prefer running their test suites against the wheels separately on each platform, but this is left as an exercise for the reader to implement.
Hopefully Travis will some day sort out their lack of Windows support.
The example configuration could be simplified somewhat once pip starts supporting pyproject.toml (included in the sample project). That should enable the removal of the "pip install wheel Cython" lines all over the configuration.
Categories: FLOSS Project Planets

Daniel Bader: Writing a Domain Specific Language (DSL) in Python

Planet Python - Mon, 2017-10-16 20:00
Writing a Domain Specific Language (DSL) in Python

Learn how to create your own Domain Specific Language with Python from scratch with this step-by-step tutorial.

A Domain Specific Language, or DSL for short, is a language that’s specialized to a particular application domain. In other words, it’s a programming language that’s used for a more specific application or use case than a general-purpose language like Python.

For example, regular expressions are a DSL. Another widely-used DSL is SQL. As you can see, DSLs run the gamut from the complex, like regular expressions, to the simple and very niche variety we’re going to create in this tutorial.

To give you an idea of how simple they can be, let’s take a sneak peek at what our DSL written in Python will look like:

# This is a comment module1 add 1 2 module2 sub 12 7 module1 print_results

With the DSL you’ll create in this tutorial you’ll be able to call Python functions and pass arguments to them using a syntax that resembles assembly language.

Blank lines or comment lines that start with “#” are ignored, just like Python. Any other line starts with the module name, then the function name followed by its arguments, separated by spaces.

As you’ll see in the course of this tutorial, even a simple language like this can offer a lot of flexibility and make your Python applications “scriptable.”

What You’ll Learn in This Tutorial

Writing a Domain Specific Language (DSL) may sound difficult—like something that’s really hard and should only be done by advanced programmers. Perhaps you haven’t heard of a DSL before. Or you’re not sure what one is.

If so, then this tutorial is for you. This isn’t a subject reserved for advanced programmers. A DSL doesn’t have to be complex or involve studying parser theory and abstract syntax trees.

We’re going to write a simple DSL in Python that’s generic in nature that uses other Python source files to do some work. It’s simple and generic for a reason. I want to show you how easy it is to use Python to write a DSL that you can adapt for your own use in your projects.

Even if you don’t have a direct use for a DSL today, you may pick up some new ideas or bits of the language that you haven’t seen before. We’ll look at:

  • dynamically importing Python modules at runtime
  • using getatttr() to access an object’s attributes
  • using variable-length function arguments and keyword arguments
  • converting strings to other data types
Defining Your Own Programming Language

Our DSL is a language that’s used to run Python code to perform some work. The work that’s done is completely arbitrary. It can be whatever you decide is appropriate to expose to the user that helps them accomplish their work. Also, the users of our DSL aren’t necessarily Python programmers. They just know that they have work to get done via our DSL.

It’s up to the user to decide what they need to accomplish and therefore write in the DSL source file. All the user knows is they have been provided a library of functionality, or commands, that they can run using the DSL.

For writing our DSL, we’ll start with the simplest implementation possible and incrementally add functionality. Each version of the source files you’ll see for Python and our DSL will have the same version suffix added to it.

So our first implementation will have the source files “dsl1.py”, “src1.dsl” and “module1.py”. The second version with additional functionality will end with “2” and so on.

In summary, we’ll end up with the following naming scheme for our files:

  • “src1.dsl” is the DSL source file that users write. This is not Python code but contains code written in our custom DSL.
  • “dsl1.py” is the Python source file that contains the implementation of our domain specific language.
  • “module1.py” contains the Python code that users will call and execute indirectly via our DSL.

If you ever get stuck, you can find the full source code for this tutorial on GitHub.

DSL Version 1: Getting Started

Let’s make this more concrete by deciding what the first version of our DSL will be able to do. What’s the simplest version we could make?

Since the users need to be able to run our Python code, they need to be able to specify the module name, function name and any arguments the function might accept. So the first version of our DSL will look like this:

# src1.dsl module1 add 1 2

Blank lines or comment lines that start with “#” are ignored, just like Python. Any other line starts with the module name, then the function name followed by its arguments, separated by spaces.

Python makes this easy by simply reading the DSL source file line by line and using string methods. Let’s do that:

# dsl1.py #!/usr/bin/env python3 import sys # The source file is the 1st argument to the script if len(sys.argv) != 2: print('usage: %s <src.dsl>' % sys.argv[0]) sys.exit(1) with open(sys.argv[1], 'r') as file: for line in file: line = line.strip() if not line or line[0] == '#': continue parts = line.split() print(parts)

Running “dsl1.py” from the command-line will lead to the following result:

$ dsl1.py src1.dsl ['module1', 'add', '1', '2']

If you’re using macOS or Linux, remember to make “dsl1.py” executable if it’s not already. This will allow you to run your application as a command-line command.

You can do this from your shell by running chmod +x dsl1.py. For Windows, it should work with a default Python installation. If you run into errors, check the Python FAQ.

With just a few lines of code, we were able to get a list of tokens from a line in our source file.  These token values, in the list “parts”, represent the module name, function name and function arguments. Now that we have these values, we can call the function in our module with its arguments.

Importing a Python Module at Runtime

But this brings up a new challenge. How do we import a module in Python if we don’t know the module name ahead of time? Typically, when we’re writing code, we know the module name we want to import and just enter import module1.

But with our DSL, we have the module name as the first item in a list as a string value. How do we use this?

The answer is we use can use importlib from the standard library to dynamically import the module at runtime. So let’s dynamically import our module next by adding the following line at the top of “dsl1.py” right under import sys:

import importlib

Before the with block you’ll want to add another line to tell Python where to import modules from:

sys.path.insert(0, '/Users/nathan/code/dsl/modules')

The sys.path.insert() line is necessary so Python knows where to find the directory that contains the modules that make up our library. Adjust this path as needed for your application so it references the directory where Python modules are saved.

Then, at the end of the file, insert the following lines of code:

mod = importlib.import_module(parts[0]) print(mod)

After making these changes, “dsl1.py” will look as follows:

# dsl1.py -- Updated #!/usr/bin/env python3 import sys import importlib # The source file is the 1st argument to the script if len(sys.argv) != 2: print('usage: %s <src.dsl>' % sys.argv[0]) sys.exit(1) sys.path.insert(0, '/Users/nathan/code/dsl/modules') with open(sys.argv[1], 'r') as file: for line in file: line = line.strip() if not line or line[0] == '#': continue parts = line.split() print(parts) mod = importlib.import_module(parts[0]) print(mod)

Now if we run “dsl1.py” from the command-line again, it will lead to the following result and printout:

$ dsl1.py src1.dsl ['module1', 'add', '1', '2'] <module 'module1' from '/Users/nathan/code/dsl/modules/module1.py'>

Great–we just imported a Python module dynamically at runtime using the importlib module from the standard library.

Additional importlib Learning Resources

To learn more about importlib and how you can benefit from using it in your programs, check out the following resources:

Invoking Code

Now that we’ve imported the module dynamically and have a reference to the module stored in a variable called mod, we can invoke (call) the specified function with its arguments. At the end of “dsl1.py”, let’s add the following line of code:

getattr(mod, parts[1])(parts[2], parts[3])

This may look a little odd. What’s happening here?

We need to get a reference to the function object in the module in order to call it. We can do this by using getattr with the module reference. This is the same idea as using import_module to dynamically get a reference to the module.

Passing the module to getattr and the name of the function returns a reference to the module’s add function object. We then call the function by using parentheses and passing the arguments along, the last two items in the list.

Remember, everything in Python is an object. And objects have attributes. So it follows that we’d be able to access a module dynamically at runtime using getattr to access its attributes. For more information, see getattr in the Python docs.

Let’s look at “module1.py”:

# module1.py def add(a, b): print(a + b)

If we run “dsl1.py src1.dsl” now, what will the output be? “3”? Let’s see:

$ dsl1.py src1.dsl ['module1', 'add', '1', '2'] <module 'module1' from '/Users/nathan/code/dsl/modules/module1.py'> 12

Wait, “12”? How did that happen? Shouldn’t the output be “3”?

This is easy to miss at first and may or may not be what you want. It depends on your application. Our arguments to the add function were strings. So Python dutifully concatenated them and returned the string “12”.

This brings us to a higher level question and something that’s more difficult. How should our DSL handle arguments of different types? What if a user needs to work with integers?

One option would be to have two add functions, e.g. add_str and add_int. add_int would convert the string parameters to integers:

print(int(a) + int(b))

Another option would be for the user to specify what types they’re working with and have that be an argument in the DSL:

module1 add int 1 2

What decisions you make in regards to your DSL’s syntax and how it functions depends on your application and what your users need to accomplish. What we’ve seen so far is, of course, a simple example, but the dynamic nature of Python is powerful.

In other words, Python’s built-in features can take you a long way; without having to write a lot of custom code. We’ll explore this more next in version 2 of our DSL.

You can find the final version of “dsl1.py” here on GitHub.

DSL Version 2: Parsing Arguments

Let’s move on to version 2 and make things more general and flexible for our users. Instead of hardcoding the arguments, we’ll let them pass any number of arguments. Let’s look at the new DSL source file:

# src2.dsl module2 add_str foo bar baz debug=1 trace=0 module2 add_num 1 2 3 type=int module2 add_num 1 2 3.0 type=float

We’ll add a function that splits the DSL arguments into an “args” list and a “kwargs” dictionary that we can pass to our module functions:

def get_args(dsl_args): """return args, kwargs""" args = [] kwargs = {} for dsl_arg in dsl_args: if '=' in dsl_arg: k, v = dsl_arg.split('=', 1) kwargs[k] = v else: args.append(dsl_arg) return args, kwargs

This get_args function we just wrote can be used as follows:

args, kwargs = get_args(parts[2:]) getattr(mod, parts[1])(*args, **kwargs)

After calling get_args, we’ll have an arguments list and a keyword arguments dictionary. All that’s left to do is change our module function signatures to accept *args and **kwargs and update our code to use the new values.

From within our module’s function, *args is a tuple and **kwargs is a dictionary. Here’s the new generalized code for “module2.py” that uses these new values:

# module2.py def add_str(*args, **kwargs): kwargs_list = ['%s=%s' % (k, kwargs[k]) for k in kwargs] print(''.join(args), ','.join(kwargs_list)) def add_num(*args, **kwargs): t = globals()['__builtins__'][kwargs['type']] print(sum(map(t, args)))

In add_str, kwargs_list is a list that’s created using a list comprehension. If you haven’t seen this before, a list comprehension creates a list using an expressive and convenient syntax.

We simply loop over the keys in the dictionary (for k in kwargs) and create a string representing each key/value pair in the dictionary. We then print the result of joining the list of arguments with an empty string and the result of joining the list of keyword arguments with “,“:

foobarbaz debug=1,trace=0

For more on list comprehensions, see this tutorial: “Comprehending Python’s Comprehensions”.

With add_num, we decided to give the user a little more power. Since they need to add numbers of specific types (int or float), we need to handle the string conversion somehow.

We call globals() to get a dictionary of references to Python’s global variables. This gives us access to the __builtins__ key/value which in turn gives us access to the classes and constructors for “int” and “float”.

This allows the user to specify the type conversion for the string values passed in our DSL source file “src2.dsl”, e.g. “type=int”. The type conversion is done in one step for all arguments in the call to map and its output is fed to sum.

The map() function takes a function and an iterable and calls the function for each item in the iterable, capturing its output. Think of it as a way of transforming a sequence of values into new values. If it’s not clear and it’s too much on one line, break it into two lines for clarity:

converted_types = map(t, args)  # t is class "int" or "float" print(sum(converted_types))

For the DSL source lines:

module2 add_num 1 2 3 type=int module2 add_num 1 2 3.0 type=float

We get the output:

6 6.0

Users can now pass any number of arguments to our functions. What I think is particularly helpful is the use of **kwargs, the keyword arguments dictionary.

Users can call our functions with keywords from the DSL, passing options, just like they’d do if they were Python programmers or running programs from the command line. Keywords are also a form of micro-documentation and serve as reminders for what’s possible. For best results, try to pick succinct and descriptive names for your keyword arguments.

Once again you can find the final version of “dsl2.py” on GitHub.

DSL Version 3: Adding Documentation

Let’s add one more feature to help our users and create version 3. They need some documentation. They need a way to discover the functionality provided by the library of modules.

We’ll add this feature by adding a new command line option in “dsl3.py” and checking the modules and their functions for docstrings. Python docstrings are string literals that appear as the first line of a module, function, class or method definition. The convention is to use triple-quoted strings like this:

def function_name(): """A helpful docstring.""" # Function body

When users pass “help=module3” on the command line to “dsl3.py”, the get_help function is called with “module3”:

def get_help(module_name): mod = importlib.import_module(module_name) print(mod.__doc__ or '') for name in dir(mod): if not name.startswith('_'): attr = getattr(mod, name) print(attr.__name__) print(attr.__doc__ or '', '\n')

In get_help, the module is dynamically imported using import_module like we’ve done before. Next we check for the presence of a docstring value using the attribute name __doc__ on the module.

Then we need to check all functions in the module for a docstring. To do this we’ll use the built-in function “dir”. “dir” returns a list of all attribute names for an object. So we can simply loop over all the attribute names in the module, filter out any private or special names that begin with “_” and print the function’s name and docstring if it exists.

The final version of “dsl3.py” is also available on GitHub.

Writing a DSL With Python – Review & Recap

Let’s recap what we’ve done in this tutorial. We’ve created a simple DSL that lets our users easily get some work done by calling into a library of functions. Luckily for us, we know Python. So we can use it to implement our DSL and make things easy for us too.

DSLs are powerful tools that are fun to think about and work on. They’re another way we can be creative and solve problems that make it easier for our users to get work done. I hope this tutorial has given you some new ideas and things to think about that you can apply and use in your own code.

From the user’s perspective, they’re just running “commands.” From our perspective, we get to leverage Python’s dynamic nature and its features and, in turn, reap the rewards of having all of the power of Python and its ecosystem available to us. For example, we can easily make changes to a library module or extend the library with new modules to expose new functionality using the standard library or 3rd party packages.

In this tutorial we looked at a few techniques:

  • importlib.import_module(): dynamically import a module at runtime
  • getattr(): get an object’s attribute
  • variable-length function arguments and keyword arguments
  • converting a string to a different type

Using just these techniques is quite powerful. I encourage you to take some time to think about how you might extend the code and functionality I’ve shown here. It could be as simple as adding a few lines of code using some of the features built-in to Python or writing more custom code using classes.

Using importlib

I’d like to mention one more thing regarding the use of “importlib”. Another application and example of using dynamic imports with “importlib” is implementing a plugin system. Plugin systems are very popular and widely used in all types of software.

There’s a reason for this. Plugin systems are a method of allowing extensibility and flexibility in an otherwise static application. If you’re interested in deepening your knowledge, see Dan’s excellent tutorial “Python Plugin System: Load Modules Dynamically With importlib“

Error Checking

In this tutorial I’ve omitted error checking on purpose. One reason is to keep additional code out of the examples for clarity. But also so the users and Python programmers of the library modules can see a full stack trace when there are errors.

This may or may not be the right behavior for your application. Think about what makes the most sense for your users and handle errors appropriately, especially for common error cases.

Security Considerations

A cautionary note on security: please consider and be aware that the dynamic nature of importing and running code may have security implications depending on your application and environment. Be sure that only authorized users have access to your source and module directories. For example, unauthorized write access to the “modules” directory will allow users to run arbitrary code.

Python DSLs: Next Steps

Where do we go from here? What’s next? You may be thinking, “Well, this is nice and all, but I need more cowbell! I need to create a real DSL with real syntax and keywords.”

A good next step would be to look at Python parsing libraries. There are many! And their functionality, ease-of-use and documentation vary widely.

If you’d like to use the code used in this tutorial for your own experiments, the full source code is available on GitHub.

Categories: FLOSS Project Planets

Justin Mason: Links for 2017-10-16

Planet Apache - Mon, 2017-10-16 19:58
Categories: FLOSS Project Planets

FSF Events: Richard Stallman - "El software libre y tu libertad" (Puebla, Mexico)

GNU Planet! - Mon, 2017-10-16 17:45
Richard Stallman hablará sobre las metas y la filosofía del movimiento del Software Libre, y el estado y la historia del sistema operativo GNU, el cual junto con el núcleo Linux, es actualmente utilizado por decenas de millones de personas en todo el mundo.

Esa charla de Richard Stallman, organizada por la Acción Directa Autogestiva, no será técnica y será abierta al público; todos están invitados a asistir.

Lugar: Karuzo, 11 Oriente 218, Centro Histórico, Puebla, México

Favor de rellenar este formulario, para que podamos contactarle acerca de eventos futuros en la región de Puebla.

Categories: FLOSS Project Planets
Syndicate content