FLOSS Project Planets

Satyam Zode: Google Summer of Code 2016 : Final Report

Planet Debian - Mon, 2016-08-22 10:02
Project Title : Improving diffoscope tool and reproducibility of Debian packages

Project details

This project aims to improve diffoscope tool and fix Debian packages which are unreproducible in Reproducible builds testing framework. diffoscope recursively unpack archives of many kinds and transform various binary formats into more human readable form to compare them. As a part of this project I worked on argument completion feature and ignoring .buildinfo feature. This project is a part of Reproducible Builds effort

Mentor and Co-Mentor
  • Jérémy Bobbio (Lunar) : Mentor
  • Reiner Herrmann (deki) : Co-Mentor
  • Holger Levsen (h01ger) : Co-Mentor
  • Mattia Rizzolo (mapreri) : Co-Mentor

Project Discussion

  • Introduction to Reproducible Builds in Debian

    First time I came to know about Reproducible Builds was during Debconf 2015. I started to get involve from the start of March 2016. At the beginning Lunar suggested me to watch the talks given on Reproducible Builds wiki. I read documentation on Reproducible Builds site and started to participate in IRC discussions on #debian-reproducible.

  • Application Review Period

    During proposal discussion period we discussed the areas where work needs to be done. I wrote the proposal and got it reviewed by community on the mailing list. Simultaneously, I worked on bug #818111 and submitted patch for same. That not only helped me to understand the concept of Reproducible Builds but also helped me to setup testing environment required to check the reproducibility of Debian packages.

  • Community Bonding Period

    During community bonding period I studied the codebase of diffoscope and also spent enough amount of time for learning Python3 metaprogramming and other OOP concepts. We also discussed more about hiding differences and options for same. I couldn’t finish my project research work during this period since I had exams in May 2016 and it consumed almost half of community bonding period and week 1 of coding period.

Project Implementation

Challenges and Work Left
  • To understand the main purpose of diffoscope in the context of Reproducible Builds. I had to go through complete Reproducible Builds project. It consumed significant amount of time to understand what Reproducible Builds is, why it’s necessary important for Free software to build reproducibly. Diffoscope is the last tool in Reproducible Builds toolchain. It was a big challenge for me to understand whole process and objective of diffoscope.
  • Work Left:

Future work
  • Based on the research work and implementation done during Coding Period make diffoscope better and enhance ignoring capabilities of diffoscope.
  • Improve the parallel processing feature of diffoscope. This particular problem is hard to understand and implement.
  • Make diffoscope better by solving exsting bugs.


I would like to express my deepest gratitude to Lunar for mentoring me throughout Google Summer of Code program and for being cool. Lunar’s deep knowledge regarding diffoscope and Python skills helped me a lot throughout the project and we literally had great discussions. I would also like to thank Debaian community and Google for giving me this opportunity. Special thanks to Reproducible Builds folks for all the guidance!

Categories: FLOSS Project Planets

DebConf team: Proposing speakers for DebConf17 (Posted by DebConf17 team)

Planet Debian - Mon, 2016-08-22 09:44

As you may already know, next DebConf will be held at Collège de Maisonneuve in Montreal from August 6 to August 12, 2017. We are already thinking about the conference schedule, and the content team is open to suggestions for invited speakers.

Priority will be given to speakers who are not regular DebConf attendees, who are more likely to bring diverse viewpoints to the conference.

Please keep in mind that some speakers may have very busy schedules and need to be booked far in advance. So, we would like to start inviting speakers in the middle of September 2016.

If you would like to suggest a speaker to invite, please follow the procedure described on the Inviting Speakers page of the DebConf wiki.

DebConf17 team

Categories: FLOSS Project Planets

Python 4 Kids: Python for Kids: Python 3 – Project 7

Planet Python - Mon, 2016-08-22 09:25

Using Python 3 in Project 7 of Python For Kids For Dummies

In this post I talk about the changes that need to be made to the code of
Project 7 in order for it to work with Python 3. Most of the code in project 7 will work without changes. However, in a lot of cases what Python outputs in Python 3 is different from the output in Python 2.7 and it’s those changes that I am mainly identifying below.


Some people want to use my book Python for Kids for Dummies to learn Python 3.
I am working through the code in the existing book, highlighting changes from Python 2 to Python 3 and providing code that will work in Python 3. If you are using Python 2.7 you can ignore this post. This post is only for people who want to take the code in my book Python for Kids for Dummies and run it in Python 3.

Page 178

All code on this page is the same, and all outputs from the code is the same in Python 3 as in Python 2.7

Page 179-180
The code and syntax on these pages is the same, but the outputs are
different in Python 3. This is because, in Python 3,
the range builtin does not create a list as in Python 2.7 (see Python3/Project 5)

#Python 2.7 code: >>> test_string = '0123456789' >>> test_string[0:1] '0' >>> test_string[1:3] '12' >>> # range(10) is a list of the numbers from 0 to 9 inclusive >>> range(10)[0:1] [0] >>> range(10)[1:3] [1, 2] >>> test_string[:3] '012' >>> test_string[3:] '3456789' #Python 3 code: >>> test_string = '0123456789' >>> test_string[0:1] '0' >>> test_string[1:3] '12' >>> # range(10) is no longer a list. It's a.... errr... range >>> # so the [:] operator slices. You can use list() >>> # to see what it corresponds to. >>> range(10)[0:1] range(0, 1) >>> list(range(10)[0:1]) [0] >>> # note same output as in Python 2.7 from range(10)[0:1] >>> range(10)[1:3] range(1, 3) >>> list(range(10)[1:3]) [1, 2] >>> test_string[:3] '012' >>> test_string[3:] '3456789' >>>

Pages 180-196
All code on this page is the same, and all outputs from the code is the same in Python 3 as in Python 2.7

Page 199

The code on this page uses raw_input, which has been renamed to input in Python 3.
Either change all occurrences or add a line

raw_input = input

at the start of the relevant code.

#Python 2.7 code: #### Input and Output Section message = raw_input("Type the message to process below:\n") ciphertext = encrypt_msg(message, ENCRYPTION_DICT) plaintext = decrypt_msg(message, DECRYPTION_DICT) print("This message encrypts to") print(ciphertext) print # just a blank line for readability print("This message decrypts to") print(plaintext) #Python 3 code: #### Input and Output Section message = input("Type the message to process below:\n") ciphertext = encrypt_msg(message, ENCRYPTION_DICT) plaintext = decrypt_msg(message, DECRYPTION_DICT) print("This message encrypts to") print(ciphertext) print # just a blank line for readability print("This message decrypts to") print(plaintext) >>> ================================== RESTART ================================ >>> Type the message you'd like to encrypt below: I love learning Python. And my teacher is smelly. And I shouldn't start a sentence with and. This message encrypts to F|ilsb|ib7okfkd|Mvqelk+|xka|jv|qb79ebo|fp|pjbiiv+||xka|F|pelriak$q|pq7oq|7|pbkqbk9b|tfqe|7ka+ This message decrypts to L2oryh2ohduqlqj2SBwkrq;2Dqg2pB2whdfkhu2lv2vphooB;22Dqg2L2vkrxogq*w2vwduw2d2vhqwhqfh2zlwk2dqg; >>> ================================== RESTART ================================ >>> Type the message you'd like to encrypt below: F|ilsb|ib7okfkd|Mvqelk+|xka|jv|qb79ebo|fp|pjbiiv+||xka|F|pelriak$q|pq7oq|7|pbkqbk9b|tfqe|7ka+ This message encrypts to C_fip8_f84lhcha_Jsnbih(_uh7_gs_n846b8l_cm_mg8ffs(__uh7_C_mbiof7h!n_mn4ln_4_m8hn8h68_qcnb_4h7( This message decrypts to I love learning Python. And my teacher is smelly. And I shouldn't start a sentence with and.

Page 200

This code works as is in both Python 2.7 and Python 3. However, the way the open() builtin works has changed in Python 3 and this will cause some issues in later projects. In Python 3 open() has the same syntax as in Python 2.7, but uses a different way to get data out of the file and into your hands. As a practical matter this means that some Python 2.7 code will sometimes cause problems when run in Python 3. If you run into such a problem (open code that works in Python 2.7 but fails in Python 3), the first thing to try is to add the binary modifier. So,
instead of ‘r’ or ‘w’ for read and write use ‘rb’ or ‘wb’. This code doesn’t need it, but a later project will.

Page 201

The code on this page is the same, but the outputs are different in Python 3. Python 3 returns how much data has
been written (in this case, 36)

#Python 2.7 code: >>> file_object = open('p4k_test.py','w') >>> text = "print('Hello from within the file')\n" # watch the " and ' >>> file_object.write(text) # writes it to the file >>> file_object.write(text) # writes it to the file again! >>> file_object.close() # finished with file, so close it #Python 3 code: >>> file_object = open('p4k_test.py','w') >>> text = "print('Hello from within the file')\n" # watch the " and ' >>> file_object.write(text) # writes it to the file 36 >>> file_object.write(text) # writes it to the file again! 36 >>> file_object.close() # finished with file, so close it

Pages 202 and 203

All code on these page is the same, and all outputs from the code is the same in Python 3 as in Python 2.7

Page 204

All code on this page is the same in Python 3 as in Python 2.7, but some of the outputs are different
A line has been added in the Python 3 code below to shown that the file_object has been closed after leaving the
with clause – this was explicit in the print out in Python 2.7.

>>> #Python 2.7 >>> with open('p4k_test.py','r') as file_object: print(file_object.read()) print('Hello from within the file') print('Hello from within the file') >>> file_object <closed file 'p4k_test.py', mode 'r' at 0xf7fed0> >>> #Python 3 >>> with open('p4k_test.py','r') as file_object: print(file_object.read()) print('Hello from within the file') print('Hello from within the file') >>> file_object # output different from 2.7 <_io.TextIOWrapper name='p4k_test.py' mode='r' encoding='UTF-8'> >>> file_object.closed # but the file is still closed True

Page 205

All code on this page is the same in Python 3 as in Python 2.7, but some of the outputs are different
A line has been added in the Python 3 code below to shown that the file_object has been closed after leaving the
with clause – this was explicit in the print out in Python 2.7. Also, because Python 3 uses a different way
of getting information from a file it is identified differently. In Python 2.7 it’s call a file – pretty straight
forward. in Python 3 it’s called a _io.TextIOWrapper. Not as enlightening, but a student doesn’t need to worry about
this difference in detail.

>>> #Python 2.7 >>> with open('testfile2','w') as a: a.write('stuff') >>> with open('testfile2','r') as a, open('p4k_test.py','r') as b: print(a.read()) print(b.read()) stuff print('Hello from within the file') print('Hello from within the file') >>> a <closed file 'testfile2', mode 'r' at 0xf6e540> >>> b <closed file 'p4k_test.py', mode 'r' at 0xef4ed0> >>> #Python 3 >>> with open('testfile2','r') as a, open('p4k_test.py','r') as b: print(a.read()) print(b.read()) stuff print('Hello from within the file') print('Hello from within the file') >>> a <_io.TextIOWrapper name='testfile2' mode='r' encoding='UTF-8'> >>> a.closed True >>> b <_io.TextIOWrapper name='p4k_test.py' mode='r' encoding='UTF-8'> >>> b.closed True

Page 207

All code on this page is the same in Python 3 as in Python 2.7, but some of the outputs are different
(the write method returns the amount of data written and this is output in the console in Python 3)

>>> #Python 2.7 >>> INPUT_FILE_NAME = "cryptopy_input.txt" >>> with open(INPUT_FILE_NAME,'w') as input_file: input_file.write('This is some test text') >>> #Python 3 >>> INPUT_FILE_NAME = "cryptopy_input.txt" >>> with open(INPUT_FILE_NAME,'w') as input_file: input_file.write('This is some test text') 22

# this code is the same in Python 2.7 and Python 3:

INPUT_FILE_NAME = “cryptopy_input.txt”
OUTPUT_FILE_NAME = “cryptopy_output.txt”

Page 208-218
All code on this page is the same, and all outputs from the code is the same in Python 3 as in Python 2.7

Categories: FLOSS Project Planets

BlackMesh: Our Ongoing Commitment to Security: Partnering with Tag1 Consulting

Planet Drupal - Mon, 2016-08-22 09:22

As you may already know, the BlackMesh team is committed to ensuring developers can focus on their website goals without worrying about scalability, infrastructure, and – in particular – security. That’s why we’re thrilled to have partnered with leading Drupal security agency Tag1 Consulting to provide our Drupal clients a comprehensive tool for managing site updates.


The new Tag1 Quo is a hosted security dashboard that provides up-to-the-minute snapshots of a client’s security status and potential vulnerabilities. Basically, this innovative tool provides Drupal users consolidated and critical security updates for their websites. Tag1 Quo features include essentials for a website’s success, such as self-service monitoring, security notifications for out-of-date modules, and patch and release delivery.


Tag1 Quo automatically monitors upstream releases and security advisories in collaboration with module maintainers and other Drupal 6 long term support (LTS) providers. The Tag1 team of Drupal security experts review and decide which issues affect the client, carefully backporting those that are applicable to their site. Quo then delivers timely notifications to the client’s inbox. Quo users can choose to either leverage pre-patched releases or quickly apply the patches themselves to bring their website up-to-date and secure against all known vulnerabilities. For extra peace of mind, the Quo dashboard provides at-a-glance visualization of all client websites, highlighting all outstanding updates across them.

Tag1 Quo Dashboard


Though D6 was phased out earlier this year, Tag1 Consulting continues to provide long-term D6 support to those who need it. As a way to adequately manage the significant maintenance and monitoring of these systems, the Tag1 team developed Tag1 Quo.

If you are currently managing a D6 site, signing up for Tag1 Quo is a no-brainer. It provides affordable long-term support for all of your core and contributed modules and themes, tested and delivered by an approved LTS provider, backed by a team of renowned Drupal experts.

Your Security. Your Options …

Even if you don’t maintain a D6 site, you’ll still want to check it out.

Depending on your needs and budget, Tag1 Quo offers three different plan options. Tag1’s roadmap includes upcoming support for D7 and D8, WordPress 4.6 and 4.7, application programming interface (API) access, upgrade planning, and more.

Signing up for Tag1 Quo means less time spent on maintenance and security – and that means more time focusing on your goals and overall mission. Whenever BlackMesh teams up with companies like Tag1, we’re advancing our commitment to your security. Contact us to learn more about how we can make the Tag1 Quo dashboard solution fit your needs.


Like this article? Follow us on Facebook and share your thoughts!

Categories: FLOSS Project Planets

Mike Driscoll: ANN: The wxPython Cookbook Kickstarter

Planet Python - Mon, 2016-08-22 09:03

Several years ago, the readers of this blog asked me to take some of my articles and turn them into a cookbook on wxPython. I have finally decided to do just that. I am including over 50 recipes that I am currently editing to make them more consistent and updating them to be compatible with the latest versions of wxPython. I currently have nearly 300 pages of content!

To help fund the initial production of the book, I am doing a fun little Kickstarter campaign for the project. The money raised will be used for the unique perks offered in the campaign as well as various production costs related to the book, such as ISBN acquisition, artwork, software expenses, advertising, etc.

In case you don’t know what wxPython is, the wxPython package is a popular toolkit for creating cross platform desktop user interfaces. It works on Windows, Mac and Linux with little to no modification of your code base.

The examples in my book will work with both wxPython 3.0.2 Classic as well as wxPython Phoenix, which is the bleeding edge of wxPython that supports Python 3. If I discover any recipes that do not work with Phoenix, they will be clearly marked or there will be an alternative example given that does work.

Here is a listing of the current set of recipes in no particular order:

  • Adding / Removing Widgets Dynamically
  • How to put a background image on a panel
  • Binding Multiple Widgets to the Same Handler
  • Catching Exceptions from Anywhere
  • wxPython’s Context Managers
  • Converting wx.DateTime to Python datetime
  • Creating an About Box
  • How to Create a Login Dialog
  • How to Create a “Dark Mode”
  • Generating a Dialog from a Config File
  • How to Disable a Wizard’s Next Button
  • How to Use Drag and Drop
  • How to Drag and Drop a File From Your App to the OS
  • How to Edit Your GUI Interactively Using reload()
  • How to Embed an Image in the Title Bar
  • Extracting XML from the RichTextCtrl
  • How to Fade-in a Frame / Dialog
  • How to Fire Multiple Event Handlers
  • Making your Frame Maximize or Full Screen
  • Using wx.Frame Styles
  • Get the Event Name Instead of an Integer
  • How to Get Children Widgets from a Sizer
  • How to Use the Clipboard
  • Catching Key and Char Events
  • Learning How Focus Works in wxPython
  • Making Your Text Flash
  • Minimizing to System Tray
  • Using ObjectListView instead of ListCtrl
  • Making a Panel Self-Destruct
  • How to Switch Between Panels
  • wxPython: Using PyDispatcher instead of Pubsub
  • Creating Graphs with PyPlot
  • Redirect Python’s Logging Module to a TextCtrl
  • Redirecting stdout / stderr
  • Resetting the Background Color
  • Saving Data to a Config File
  • How to Take a Screenshot of Your wxPython App and Print it
  • Creating a Simple Notebook
  • Ensuring Only One Instance Per Frame
  • Storing Objects in ComboBox or ListBox Widgets
  • Syncing Scrolling Between Two Grids
  • Creating Taskbar Icons
  • A wx.Timer Tutorial
  • How to Update a Progress Bar from a Thread
  • Updating Your Application with Esky
  • Creating a URL Shortener
  • Using Threads in wxPython
  • How to Create a Grid in XRC
  • An Introduction to XRC

 Note: Recipe names and order are subject to change

Categories: FLOSS Project Planets

Doug Hellmann: random — Pseudorandom Number Generators — PyMOTW 3

Planet Python - Mon, 2016-08-22 09:00
The random module provides a fast pseudorandom number generator based on the Mersenne Twister algorithm. Originally developed to produce inputs for Monte Carlo simulations, Mersenne Twister generates numbers with nearly uniform distribution and a large period, making it suited for a wide range of applications. Read more… This post is part of the Python Module … Continue reading random — Pseudorandom Number Generators — PyMOTW 3
Categories: FLOSS Project Planets

Mike Driscoll: PyDev of the Week: Michele Simionato

Planet Python - Mon, 2016-08-22 08:30

This week we welcome Michele Simionato as our PyDev of the Week! Michele is an expert on Python and is known for his paper on Python’s Method Resolution Order which was published to the Python website by Guide Van Rossum and for a very interesting series of articles on metaclasses that he wrote with David Mertz. They are a bit difficult to find, but you can read the first one of the 3-part series here. He is one of the founders of the Italian Python Association. Michele has a Ph. D. about the Renormalization of Quantum Field Theory. Let’s take a few moments

Can you tell us a little about yourself (hobbies, education, etc):

I originally come from academia and I have a Ph. D. in Theoretical Physics. Then I worked for several years for an Analytics firm (stock market risk assessment) and now I am back to science, doing earthquake simulations at GEM.

Why did you start using Python?

It happened in 2002. At the time I was a postdoc researcher in the department of Physics and Astronomy at Pittsburgh University. I decided that it was time to learn some modern programming language, in view of a possible career outside academia. After reading a couple of long books by Bruce Eckel, first about C++ and then about Java, I decided that I did not want to program in either of them. I was in doubt between Ruby and Python, but Python won because of the better scientific libraries and of the more pragmatic philosophy.

What other programming languages do you know and which is your favorite?

A long time ago I started with Basic and Pascal and later on I worked a lot with Mathematica and Maple. After learning Python I become interested in functional languages and I know decently well Scheme, so much that I nearly wrote a book on it, The Adventures of a Pythonista in Schemeland. In my daily job I had to work a lot with SQL (which I like enough) and with Javascript (which I don’t like).

What projects are you working on now?

In the last three years I have become the maintainer and the main developer of the OpenQuake Engine, which is a computational engine to produce earthquake hazard and risk assessment. It means that after several years of being a database and Web developer I have become a scientific programmer and now I spend most of my time doing performance analysis of massive distributed calculations. I also keep a blog where I document my fighting with the engine.

Which Python libraries are your favorite (core or 3rd party)?

numpy is a really well thought library, an essential tool for people doing scientific applications.

Where do you see Python going as a programming language?

Honestly, I am unsure about where Python as a language is going, and I am not even convinced I like the recent trend. Certainly I would like for the language to become simpler, that’s what attracted me to Python in the first place, and instead I see several things that are becoming increasingly complicated. Also, there are now other languages out there that are worth of note, whereas for years Python had no competitors. If you want to know, I am thinking about Go for server side programming and about Julia for scientific programming. Both of them looks really interesting even if I have not programmed in either of them. Python should not rest thinking that it is best than Java and C++ (an easy win) and instead consider seriously the new contenders.

What is your take on the current market for Python programmers?

It has always been a good market for Python programmers (at least from when I started, 14 years ago) and now it is even more so. I get offers for Python jobs nearly every week.

Is there anything else you’d like to say?

My tagline at the EuroPython 2016 conference was “Legacy code warrior”: that reflects my daily job in the last 10 years at least. You can see a video of my talk here:

Thanks for doing the interview!

Categories: FLOSS Project Planets

Hynek Schlawack: Better Python Object Serialization

Planet Python - Mon, 2016-08-22 08:30

The Python standard library is full of underappreciated gems. One of them allows for simple and elegant function dispatching based on argument types. This makes it perfect for serialization of arbitrary objects – for example to JSON in web APIs and structured logs.

Categories: FLOSS Project Planets

Vincent Sanders: Down the rabbit hole

Planet Debian - Mon, 2016-08-22 08:24
My descent began with a user reporting a bug and I fear I am still on my way down.

The bug was simple enough, a windows bitmap file caused NetSurf to crash. Pretty quickly this was tracked down to the libnsbmp library attempting to decode the file. As to why we have a heavily used library for bitmaps? I am afraid they are part of every icon file and many websites still have favicons using that format.

Some time with a hex editor and the file format specification soon showed that the image in question was malformed and had a bad offset header entry. So I was faced with two issues, firstly that the decoder crashed when presented with badly encoded data and secondly that it failed to deal with incorrect header data.

This is typical of bug reports from real users, the obvious issues have already been encountered by the developers and unit tests formed to prevent them, what remains is harder to produce. After a debugging session with Valgrind and electric fence I discovered the crash was actually caused by running off the front of an allocated block due to an incorrect bounds check. Fixing the bounds check was simple enough as was working round the bad header value and after adding a unit test for the issue I almost moved on.


We already used the bitmap test suite of images to check the library decode which was giving us a good 75% or so line coverage (I long ago added coverage testing to our CI system) but I wondered if there was a test set that might increase the coverage and perhaps exercise some more of the bounds checking code. A bit of searching turned up the american fuzzy lop (AFL) projects synthetic corpora of bmp and ico images.

After checking with the AFL authors that the images were usable in our project I added them to our test corpus and discovered a whole heap of trouble. After fixing more bounds checks and signed issues I finally had a library I was pretty sure was solid with over 85% test coverage.

Then I had the idea of actually running AFL on the library. I had been avoiding this because my previous experimentation with other fuzzing utilities had been utter frustration and very poor return on investment of time. Following the quick start guide looked straightforward enough so I thought I would spend a short amount of time and maybe I would learn a useful tool.

I downloaded the AFL source and built it with a simple make which was an encouraging start. The library was compiled in debug mode with AFL instrumentation simply by changing the compiler and linker environment variables.

$ LD=afl-gcc CC=afl-gcc AFL_HARDEN=1 make VARIANT=debug test
afl-cc 2.32b by <lcamtuf@google.com>
afl-cc 2.32b by <lcamtuf@google.com>
COMPILE: src/libnsbmp.c
afl-cc 2.32b by <lcamtuf@google.com>
afl-as 2.32b by <lcamtuf@google.com>
[+] Instrumented 751 locations (64-bit, hardened mode, ratio 100%).
AR: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/libnsbmp.a
COMPILE: test/decode_bmp.c
afl-cc 2.32b by <lcamtuf@google.com>
afl-as 2.32b by <lcamtuf@google.com>
[+] Instrumented 52 locations (64-bit, hardened mode, ratio 100%).
LINK: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp
afl-cc 2.32b by <lcamtuf@google.com>
COMPILE: test/decode_ico.c
afl-cc 2.32b by <lcamtuf@google.com>
afl-as 2.32b by <lcamtuf@google.com>
[+] Instrumented 65 locations (64-bit, hardened mode, ratio 100%).
LINK: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_ico
afl-cc 2.32b by <lcamtuf@google.com>
Test bitmap decode
Tests:606 Pass:606 Error:0
Test icon decode
Tests:392 Pass:392 Error:0
TEST: Testing complete
I stuffed the AFL build directory on the end of my PATH, created a directory for the output and ran afl-fuzz

afl-fuzz -i test/bmp -o findings_dir -- ./build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp @@ /dev/null
The result was immediate and not a little worrying, within seconds there were crashes and lots of them! Over the next couple of hours I watched as the unique crash total climbed into the triple digits.

I was forced to abort the run at this point as, despite clear warnings in the AFL documentation of the demands of the tool, my laptop was clearly not cut out to do this kind of work and had become distressingly hot.

AFL has a visualisation tool so you can see what kind of progress it is making which produced a graph that showed just how fast it managed to produce crashes and how much the return plateaus after just a few cycles. Although it was finding a new unique crash every ten minutes or so when aborted.

I dove in to analyse the crashes and it immediately became obvious the main issue was caused when the test tool attempted allocations of absurdly large bitmaps. The browser itself uses a heuristic to determine the maximum image size based on used memory and several other values. I simply applied an upper bound of 48 megabytes per decoded image which fits easily within the fuzzers default heap limit of 50 megabytes.

The main source of "hangs" also came from large allocations so once the test was fixed afl-fuzz was re-run with a timeout parameter set to 100ms. This time after several minutes no crashes and only a single hang were found which came as a great relief, at which point my laptop had a hard shutdown due to thermal event!

Once the laptop cooled down I spooled up a more appropriate system to perform this kind of work a 24way 2.1GHz Xeon system. A Debian Jessie guest vm with 20 processors and 20 gigabytes of memory was created and the build replicated and instrumented.

To fully utilise this system the next test run would utilise AFL in parallel mode. In this mode there is a single "master" running all the deterministic checks and many "secondary" instances performing random tweaks.

If I have one tiny annoyance with AFL, it is that breeding and feeding a herd of rabbits by hand is annoying and something I would like to see a convenience utility for.

The warren was left overnight with 19 instances and by morning had generated crashes again. This time though the crashes actually appeared to be real failures.

$ afl-whatsup sync_dir/
Summary stats

Fuzzers alive : 19
Total run time : 5 days, 12 hours
Total execs : 214 million
Cumulative speed : 8317 execs/sec
Pending paths : 0 faves, 542 total
Pending per fuzzer : 0 faves, 28 total (on average)
Crashes found : 554 locally unique
All the crashing test cases are available and a simple file command immediately showed that all the crashing test files had one thing in common the height of the image was -2147483648 This seemingly odd number is actually meaningful to a programmer, it is the largest negative number which can be stored in a 32bit integer (INT32_MIN) I immediately examined the source code that processes the height in the image header.

if ((width <= 0) || (height == 0))
if (height < 0) {
bmp->reversed = true;
height = -height;
The bug is where the height is made a positive number and results in height being set to 0 after the existing check for zero and results in a crash later in execution. A simple fix was applied and test case added removing the crash and any possible future failure due to this.

Another AFL run has been started and after a few hours has yet to find a crash or non false positive hang so it looks like if there are any more crashes to find they are much harder to uncover.

Main lessons learned are:
  • AFL is an easy to use and immensely powerful and effective tool. State of the art has taken a massive step forward.
  • The test harness is part of the test! make sure it does not behave in a poor manner and cause issues itself.
  • Even a library with extensive test coverage and real world users can benefit from this technique. But it remains to be seen how quickly the rate of return will reduce after the initial fixes.
  • Use the right tool for the job! Ensure you head the warnings in the manual as AFL uses a lot of resources including CPU, disc and memory.
I will of course be debugging any new crashes that occur and perhaps turning my sights to all the projects other unit tested libraries. I will also be investigating the generation of our own custom test corpus from AFL to replace the demo set, this will hopefully increase our unit test coverage even further.

Overall this has been my first successful use of a fuzzing tool and a very positive experience. I would wholeheartedly recommend using AFL to find errors and perhaps even integrate as part of a CI system.
Categories: FLOSS Project Planets

"Menno's Musings": IMAPClient 1.0.2

Planet Python - Mon, 2016-08-22 08:14

IMAPClient 1.0.2 is out! This is release comes with a few small fixes and tweaks, as well as a some documentation improvements.


  • There's now an explicit check that the pyOpenSSL version that IMAPClient is seeing is sufficient. This is to help with situations (typically on OS X) where the (old) system pyOpenSSL takes precedence over the version that IMAPClient needs. Use of virtualenvs is highly recommended.
  • Python 3.5 is now officially supported and tested against.
  • setup.py can now be used even if it's not in the current directory.
  • Handling of RFC2822 group address syntax has been documented.
  • The INI file format used by the live tests and interactive shell has finally been documented.
  • Links to ReadTheDocs now go to readthedocs.io
  • The project README has been arranged so that all the essentials are right at the top.

I announced that the project would be moving to Git and Github some time ago and this is finally happening. This release will be the last release where the project is on Bitbucket.

Categories: FLOSS Project Planets

Python Software Foundation: "In the beginning, there was one Python group": Community Service Award Recipient Stéphane Wirtel

Planet Python - Mon, 2016-08-22 06:30
“In the beginning, there was one Python group in Charleroi, the P3B (Python Blanc Bleu Belge)”, Stéphane Wirtel recalls. This first Python group was led by Denis Frère and Olivier Laurent. Together with Aragne, the first company using Python in Belgium, and Marc-Andre Lemburg the P3B helped organize the inaugural EuroPython in 2002. Over the years, however, the P3B disbanded. “Other groups have organized some events for the Belgian community”, Wirtel adds. These groups, however, have faced some of the organizing challenges as the P3B.

As a Python user of 15 years, Wirtel contemplated what would be the best way to sustainably build the Belgian Python community. He originally wanted to organize the first PyCon in Belgium but eventually decided to invest his energies elsewhere. Ludovic Gasc, Fabien Benetou and Wirtel began by hosting Python events in Brussels and Charleroi.

The Python Software Foundation has awarded Wirtel in the second quarter of 2016 with a Community Service Award in recognition for his work organizing a Python User Group in Belgium, for his continued work creating marketing material for the PSF, for his continued outreach efforts with spreading the PSF's mission.

Outreach at PythonFOSDEM and Building a New Python Belgium Community

“FOSDEM is one of the most important events in the European development community with over 5,000 attendees participating in a weekend event” Wirtel explains. The importance of FOSDEM led Wirtel and Gasc to create the first PythonFOSDEM.

Since 2013 Wirtel has organized the PythonFOSDEM devroom, expanding the room from 80 participants in 2013 to well over 400 participants in 2016. Benetou, who volunteered in the FOSDEM 2016 Python devroom, remembers the excitement in the room explaining that the room was filled within five minutes of opening.
Python devroom at @fosdem, it didn't even started and it's packed already! pic.twitter.com/MRtzFt24ez — utopiah (@utopiah) January 30, 2016

With the growth of the PythonFOSDEM devroom and the return of AFPyro-BE, led by Ludovic Gasc, Wirtel has been focusing efforts on building the belgium@python.org mailing list and registering a Belgian Python website. “Stéphane continues to challenge us to organize bigger and bigger events”, Gasc comments on Wirtel. His continued work promoting Python in Belgium is helping provide the building blocks for a new Python community in Belgium.

Python Software Foundation Marketing Work Group

As a member of the PSF marketing work group, Wirtel is an ongoing voice in the discussion and creation of PSF marketing materials. Wirtel helped with flyer development and distribution for  PythonFOSDEM 2015, PyCon North America 2015 and PyCon Ireland 2015.

Inspiring new CPython contributors at EuroPython 2016

Wirtel spoke at EuroPython this year on the topic of CPython. His talk, titled “Exploring our Python Interpreter”, outlined the basics of how the Python interpreter works. Of notable importance Wirtel framed his talk for CPython novices, pointing out documentation on where to get started and resources for how to find CPython core mentors. Wirtel also pointed to a CPython patch he recently submitted for the __ltrace__ feature. With his patch you can compile Python to easily show the Python bytecode generated, a significant suggestion for beginners to be able to play with in the Python interpreter. Here is an example of his feature in action:

>>> __ltrace__ = None  # To enable tracing >>> print("hello")     # Now, shows bytecodes run 0: LOAD_NAME, 0 push <built-in function print> 2: LOAD_CONST, 0 push 'hello' 4: CALL_FUNCTION, 1 ext_pop 'hello' hello ext_pop <built-in function print> push None 6: PRINT_EXPR pop None 8: LOAD_CONST, 1 push None 10: RETURN_VALUE pop None

Some of Wirtel’s other projects includes working as a former core developer of Odoo from 2008 to 2014, an open source enterprise resource planner which is built with PostgreSQL and CPython. He has contributed to Gunicorn and is working to contribute more to CPython. Wirtel is also a member of the EuroPython Society and the Association Francophone de Python (AFPy) as well as a PSF Fellow. Wirtel has supported EuroPython the last two years as a volunteer and as a working group member too.
Wirtel’s passion for bringing new Pythonistas into the fold, be it through the creation and continued organizing of the PythonFOSDEM Devroom or the proliferation of CPython knowledge and tools particularly suited for the beginner, is profound. As he noted in his EuroPython 2016 talk, he was completely new to CPython at the 2014 PyCon North America at Montreal! “Simply put Wirtel is the type of person who gets things done” Benetou says, adding that “these are the type of people that inspire me, that I like”.
Categories: FLOSS Project Planets

Michal &#268;iha&#345;: Continuous integration on multiple platforms

Planet Debian - Mon, 2016-08-22 06:00

Over the weekend I've played with continuous integration for Gammu to make it run on more platforms. I had to remember many things from the Windows world on the way and the solution is not yet complete, but the basic build is working, the only problematic part are external dependencies.

First of all we already have Linux builds on Travis CI. These cover compilation with both GCC and Clang compilers, hopefully covering most of the possible problems.

Recently I've added OS X builds on Travis CI, what was pretty much painless and worked out of the box.

The next major architecture to support is Windows. Once I've discovered AppVeyor I thought it might be the way to go. The have free plans for open-source projects (though it has only one parallel build compared to four provided by Travis CI).

As our build system is cross platform based on CMake, it should work pretty much out of the box, right? Well almost, tweaking the basics took some time (unfortunately there is no CMake support on AppVeyor, so you have to script it a bit).

The most painful things on the way:

  • finding our correct way to invoke build and testsuite
  • our code was broken on Windows, making the testsuite to fail
  • how to work with power shell (no, I'm not going to like it)
  • how to download and install executable to PATH
  • test output integration with AppVeyor - done using XSLT transformation and uploading test results manually
  • 32-bit / 64-bit mess, CMake happily finds 32-bit libs during the 64-bit build and vice versa, what makes the build fail later when linking - fixed by trying if code can be built with given library
  • 64-bit code crashes in dummy driver, causing testsuite failures (this has to be something Windows specific as the code works fine on 64-bit Linux) - this seems to be caused by too big allocations on stack, moving them to heap will fix this

You can check our current appveyor.yml in case you're going to try something similar. Current build results are on AppVeyor.

As a nice side effect, we now have up to date Windows binaries for Gammu.

Filed under: Debian English Gammu | 0 comments

Categories: FLOSS Project Planets

NOKUBI Takatsugu: The 9th typhoon looks like Debian swirl logo

Planet Debian - Mon, 2016-08-22 05:01

According to my follower’s tweet:

@kazken3 台風画像と水平反転したDebianマークが一致.. pic.twitter.com/ymBoRGz9ew

— kuromabo_(:3」∠)_ (@kuromabo) 2016年8月22日

The typhoon image and horizontal flipped Debian logo looks same.

Categories: FLOSS Project Planets

Zlatan Todorić: When you wake up with a feeling

Planet Debian - Mon, 2016-08-22 02:45

I woke up at 5am. Somehow made myself to soon go back to sleep again. Woke up at 6am. Such is the life of jet-lag. Or I am just getting old for it.

But the truth wouldn't be complete with only those assertion. I woke inspired and tired and the same time. Tired because I am doing very time consumable things. Also in the same time very emotional things. AND at the exact same time things that inspire me.

On paper, I am technical leader of Purism. In reality, I have insanely good relations with my CEO for such a short time. So good that I for months were not leading the technical shift only, but also I overtook operations (getting orders and delivering them while working with our assembly line to automate most of the tasks in this field). I was playing also as first line of technical support (forums, IRC and email). Actually I was pretty much the only line of support for few months. I was doing some website changes: change some wording, updating bunch of plugins and making it sure all works, resolved (hopefully) Tor and Cloudflare issues for it, annoying caching system for forums, stopped forum spam and so on. I worked on better messaging for Purism public relations. I thought my team to use keys for signing and encryption. I interviewed (and read all mails) for people that were interested in working or helping Purism. In process of doing all that, I maybe wasn't the most speedy person for all our users needs but I hope they understand and forgive me.

I was doing all that while I was researching and developing tablets (which ended up not being the most successful campaign but we now do have them as product). I was doing all that while seeing (and resolving) that our kernel builds were failing. Worked on pushing touchpad (not so good but we are still working on) patches upstream (and they ended being upstreamed). While seeing repos being down because of our host. Repos being down because of broken sync with Debian. Repos being down because of our key mis-management. Metadata not working well. PureBrowser getting broken all the time. Tor browser out of date. No real ISO updates. Wrong sources.list entries and so on.

And the hardest part on work was, I was doing all this with very limited scope and even more limited resources. So what kept me on, what is pushing me forward and what am I doing?

One philosophy - Free software. Let me not explain it as a technical debt. Let me explain it as social movement. In age, where people are "bombed" by media, by all-time lying politicians (which use fear of non-existent threats/terror as model to control population), in age where proprietary corporations are selling your freedom so you can gain temporary convenience the term Free software is like Giordano Bruno in age of Inquisitions. Free software does not only preserve your Freedom to software source usage but it preserves your Freedom to think and think out of the box and not being punished for that. It preserves the Freedom to live - to choose what and when to do, without having the negative impact on your or others people lives. The Freedom to be transparent and to share. Because not only ideas grow with sharing, but we, as human beings, grow as we share. The Freedom to say "NO".

NO. I somehow learnt, and personally think, that the Freedom to say NO is the most important Freedom in our lives. No I will not obey some artificially created master that think they can plan and choose my life decision. No I will not negotiate my Freedom for your convenience (also, such Freedom is anyway not real and it is matter of time where you will be blown away by such illusion). No I will not accept your credit because it has STRINGS attached to it which you either don't present or you blur it in mountain of superficial wording. No I will not implant a chip inside me for sake of your research or my convenience. No I will not have social account on media where majority of people are. No, I will not have pacemaker which is a blackbox with proprietary (buggy) software and it harvesting my data without me being able to look at it.

Yin-Yang. Yes, I want to collaborate on making world better place for us all. I don't agree with most of people, but that doesn't make them my enemies (although media would like us to feel and think like that). I will try to preserve everyones Freedom as much as I can. Yes I will share with my community and friends. Yes I want to learn from better than I am. Yes I want to have awesome mentors. Yes, I will try to be awesome mentor. Yes, I choose to care and not ignore facts and actions done by me and other people. Yes, I have the right to be imperfect and do mistakes as long as I will aknowledge and work on them. Bugfixing ourselves as humans is the most important task in our lives. As in software, it is very time consumable but also as in software, it is improvement and incredible satisfaction to see better version of yourself, getting more and more features (even if that sometimes means actually getting read of other/bad features).

This all is blending with my work at Purism. I spend a lot of time thinking about projects, development and future. I must do that in order not to make grave mistakes. Failing hardware and software is not grave mistake. Serious, but not grave. Grave is if we betray ourselves and our community in pursue for Freedom. We are trying to unify many things - we want to give you security, privacy and FREEDOM with convenience. So I am pushing myself out of comfort zones and also out of conventional and sometimes even my standard way of thinking. I have seen that non-existing infrastructure for PureOS is hurting is a lot but I needed to cope with it to the time where I will be able to say: not anymore, we are starting to build our own infrastructure. I was coping with Cloudflare being assholes to Tor users but now we also shifting away from them. I came to team where people didn't properly understand what and why are we building this. Came to very small and not that efficient team.

Now, we employed a dedicated and hard working person on operations (Goran) which I trust. We have dedicated support person (Mladen) which tries hard to work with people. A very creative visual mastermind (Francois). We have a capable Debian Developer (Matthias Klumpp) working on PureOS new infra. We have a capable and dedicated sysadmins (Theo and Stelio) which we didn't even have in past. We are trying to LEVEL UP Free software and unify them in convenient solution which is lead by Joey Hess. We have a hard-working PureOS developer (Hema) who is coping with current non-existent PureOS infra. We have GNOME Boards of Directors person (Jeff) who is trying to light up our image in world (working with James, to try bring some lights into our shadows caused by infinite supply chain delays). We have created Advisory Board for Freedom, Privacy and Security which I don't want to name now as we are preparing to announce soon that (and trust me, we have good people in here).

But, the most important thing here is not that they are all capable or cool people. It is the core value in all of them - they care about Freedom and I trust them on their paths. The trust is always important but in Purism it is essential for our work. I built the workflow without time management (everyone spends their time every single day as they see it fit as long as the work gets done). And we don't create insane short deadlines because everyone else thinks it is important (and rarely something is more important than our time freedom). So the trust is built out of knowledge and the knowledge I have about them and their works is because we freely share with no strings attached.

Because of them, and other good people from our community I have the energy to sacrifice my entire time for Purism. It is not white and black: CEO and me don't always agree, some members of my team don't always agree with me or I with them, some people in community are very rude, impolite and don't respect our work but even with disagreement everyone in Purism finds agreement at the end (we use facts in our judgments) and all the people who just try to disturb my and mine teams work aren't as efficient as all the lovely words of people who believe in us, who send us words of support and who share ideas and their thoughts with us. There is no more satisfaction for me than reading a personal mail giving us kudos for the work and their understanding of underlaying amount of work and issues.

While we are limited with resources we had an occasional outcry from community to help us. Now I want to help them to help me (you see the Freedom of sharing here?). PureOS has now a wiki. It will be a community wiki which is endorsed by Purism as company. Yes you read it right, Purism considers its community part of company (you don't need to get paycheck to be Purism member). That is why a call upon contributors (technical but mostly non-technical too) to help us make PureOS wiki the best resource on net for our needs. Write tutorials for others, gather and put info on wiki, create an ideas page and vote on them so we can see what community wants to see, chat with us so we all understand what, why and how are we working on things. Make it as transparent as possible. Everyone interested please get in touch with our teams by either poking us online (IRC, social accounts) or via emails (our personal or [hr, pr, feedback]@puri.sm.

To finish this writing (as it is 8am here and I still want to rest a bit because I will have meetings for 6 hours straight today) - I wanted to share some personal insight into few things from my point of view. I wanted to say despite all the troubles and people who tried to make our time even harder (and it is already hard by all the limitation which come naturally today with our kind of work), we still create products, we still ship them, we still improved step by step, we still hired and we are still building. Keeping all that together and making progress is for me a milestone greater than just creating a technical product. I just hope we will continue and improve our pace so we can start progressing towards my personal great goal - integrate and cooperate with most of FLOSS ecosystem.

P.S. yes, I also (finally!) became an official Debian Developer - still didn't have time to sit and properly think and cry (as every good men) about it.

Categories: FLOSS Project Planets

Blair Wadman: How to create a custom block and assigning to a region in Drupal 8

Planet Drupal - Mon, 2016-08-22 02:12

One of the many changes in Drupal 8 is adding a block to a region. The block interface has been pretty consistent over the years, so changes to how it works can be confusing at first. You do something over and over again and then “Wait a minute! Things have moved, what do I do?!”. But never fear, the new way of adding blocks to regions is pretty straight forward once you get your head around it.

Categories: FLOSS Project Planets

Christian Perrier: [LIFE] Running activities - Ultra Trail du Mont-Blanc

Planet Debian - Mon, 2016-08-22 01:47
Hello dear readers,

It's been ages since I last blogged. Being far less active in Debian than I've been in the past, I guess this is a logical consequence.

However, I'm still active as you may witness if you read the debian-boot mailing list : I still consider myself part of the D-I team and I'm maintaining a few sports-related packages.

Most know what has taken precedence over Debian development, namely trail and ultra-trail running. And, well, it hasn't decreased, far from that : I ran about 10 races already this year....6 of them being above 50km and I ran my favourite 100km moutain race in early July for the second year in a row.

So, the upcoming week, I'll be trying to reach what is usually considered as the Grail of ultra-trail runners : the Ultra-Trail du Mont-Blanc race in Chamonix.

The race is fairly simple : run all around the Mont-Blanc summits, for a 160km race with a bit less than 10,000 meters positive climb. The race itself takes place between 800 and 2700 meters (so no "high mountain") and I expect to complete it (if I succeed) in about 40 hours.

I'm very confident (maybe too much?) as I successfully completed a much more difficult race last year (only 144km, but over 11,000 meters positive climb and a much more difficult path...it took me over 50 hours to complete it).

You can follow me on the live tracking site. The race starts on Friday August 26th, 18:00 CET DST.

I everything goes well, I have great projects for next year, including a 100-mile race in Colorado in August (we'll be traveling in USA for over 3 weeks, peaking with the solar eclipse of August 21st in Kansas City).

Categories: FLOSS Project Planets

Will Kahn-Greene: pyvideo last thoughts

Planet Python - Mon, 2016-08-22 00:00
What is pyvideo?

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

This is my last update. pyvideo.org is now in new and better hands and will continue going forward.

Read more… (2 mins to read)

Categories: FLOSS Project Planets

Full Stack Python: Python for Entrepreneurs

Planet Python - Mon, 2016-08-22 00:00

Python for Entrepreneurs is a new video course by the creators of Talk Python to Me and Full Stack Python.

We are creating this course and running a Kickstarter for it based on feedback that it's still too damn difficult to turn basic Python programming knowledge into a business to generate income as a side or full time project. Both Michael and I have been able to make that happen for ourselves and we want to share every difficult lesson we've learned through this course.

The Python for Entrepreneurs videos and content will dive into building and deploying a real-world web application, marketing it to prospective customers, handling search engine optimization, making money through credit card payments, getting help from part-time contractors for niche tasks and scaling up to meet traffic demands.

If this course hits the mark for what you want to do with Python, check out the Kickstarter - we've set up steep discounts for early backers.

Update: only 3 days left until the close of the Kickstarter on August 25 at 9am Pacific Time!

If you have any questions, please reach out to Michael Kennedy or me, Matt Makai.

Categories: FLOSS Project Planets

Paul Tagliamonte: go-wmata - golang bindings to the DC metro system

Planet Debian - Sun, 2016-08-21 22:16

A few weeks ago, I hacked up go-wmata, some golang bindings to the WMATA API. This is super handy if you are in the DC area, and want to interface to the WMATA data.

As a proof of concept, I wrote a yo bot called @WMATA, where it returns the closest station if you Yo it your location. For hilarity, feel free to Yo it from outside DC.

For added fun, and puns, I wrote a dbus proxy for the API as weel, at wmata-dbus, so you can query the next train over dbus. One thought was to make a GNOME Shell extension to tell me when the next train is. I’d love help with this (or pointers on how to learn how to do this right).

Categories: FLOSS Project Planets

PreviousNext: Drupal 8 FTW: Is it a test or is it a form? Actually, its both

Planet Drupal - Sun, 2016-08-21 21:25

As you'd be aware by now - Drupal 8 features lots of refactoring of from procedural code to object-oriented.

One such refactoring was the way forms are build, validated and executed.

One cool side-effect of this is that you can now build and test a form with a single class.

Yep that's right, the form and the test are one and the same - read on to find out more.

Categories: FLOSS Project Planets
Syndicate content