Talk Python to Me: #64 Inside the Python Package Index

Planet Python - Fri, 2016-06-24 04:00
What is the most powerful part of the Python ecosystem? Well, the ability to say "pip install magic_library" has to be right near the top. But do you what powers the Python Package Index and the people behind it? Did you know it does over 300 TB traffic each month these days? <br/> <br/> Just me as we chat with Donald Stufft to look inside Python's package infrastructure. <br/> <br/> Links from the show: <br/> <div style="font-size: .85em;"> <br/> <b>Donald on Twitter</b>: <a href='https://twitter.com/dstufft' target='_blank'>@dstufft</a> <br/> <b>Donald on the web</b>: <a href='https://caremad.io' target='_blank'>caremad.io</a> <br/> <b>Powering the Python Package Index</b>: <br/> <a href='https://caremad.io/2016/05/powering-pypi/' target='_blank'>caremad.io/2016/05/powering-pypi/</a> <br/> <b>A Year of PyPI Downloads</b>: <br/> <a href='https://caremad.io/2015/04/a-year-of-pypi-downloads/' target='_blank'>caremad.io/2015/04/a-year-of-pypi-downloads</a> <br/> <b>Donate to PPA</b>: <a href='http://donate.pypi.io/' target='_blank'>donate.pypi.io</a> <br/> <b>PyPI (Legacy)</b>: <a href='https://pypi.python.org/pypi' target='_blank'>pypi.python.org/pypi</a> <br/> <b>Warehouse (new PyPI)</b>: <a href='https://pypi.io/' target='_blank'>pypi.io</a> <br/> <b>BigQuery Data Source</b>: <br/> <a href='https://mail.python.org/pipermail/distutils-sig/2016-May/028986.html' target='_blank'>mail.python.org/pipermail/distutils-sig/2016-May/028986.html</a> <br/> </div>
Categories: FLOSS Project Planets

Gsoc 2016 Neverland #4 #5

Planet KDE - Fri, 2016-06-24 02:50

These passed two weeks were so stressful. My plan for mid-term is Neverland can build a WordPress theme.

As I mentioned in Gsoc 2016 Neverland #3 I have to choose between Sage or Underscore.
Sage is DRY but it also means that designers must seperate their theme into many smaller parts. And that also means we have more works to do.
So I choose Underscore. It’s not DRY, it’s just skeleton theme so I can adopt quickly. And one more reason I choose it because it is from .

There is a pitfall that Underscore(_s) require you rename all _s tag to your theme name https://github.com/automattic/_s#getting-started. I search for a underscore theme generator but there aren’t any promised packages.

Now the hardest part comes. I was thinking so much about how a WP theme is built from static html. Should I wrap the html with WP code or should I inject the html into WP code. You can think it’s not different. But it’s so much different when you actually code.
After tried and failed, I came up with a solution. I used mustache tags for both _s and html files. I used sublime and replace all the _s tags with mustache tags {{themeName}}. After that I deleted most of code in _s replace it with some tags like {{{ body }}},{{{content}}} … In blueprint files, I defines some tags like {{{article__author}}},{{{article__author--link}}} … (I follow BEM methodlogy). So basically rendered steps are like this:

  1. Neverland replaces {{themeName}} with the theme Name and copy all files from workers to buildings folder.
  2. Neverland renders {{{ body }}},{{{content}}} … with the corresponding files body.mustache,content.mustache
  3. Finally Neverland renders {{{article__author}}}, {{{article__author--link}}} … with WP tags like <?= get_the_author() ?>, <?= get_the_author_link() ?> …

With this approach the designers have to care about the tags {{{article__author}}} … In the future, maybe I will use DOM parser to identify the content automatically so the designers dont have to know the tags.

And here is the results:

HTML theme

WP theme – Homepage

WP theme – Single post page

As you can see, it is working. But there are still many works to do.

Thanks for reading.

Categories: FLOSS Project Planets

Norbert Preining: Rest in peace UK

Planet Debian - Fri, 2016-06-24 00:22

I am mourning for the UK. I feel so much pain and pity for all my good friends over there. Stupidity has won again. Good bye UK, your long reign has found its end. The rest is silence.


(Graphic from The Guardian – EU referendum results in full)

Categories: FLOSS Project Planets

Nigel Babu: Scraping the Indian Judicial System

Planet Python - Fri, 2016-06-24 00:20

This blog post has been sitting in my drafts folder for a long time. It’s time I finished it. A while ago, I did some work for Vidhi, scraping the Supreme Court of India website. Later on, I started some of parts of the work to scrape a couple of High Courts. Here’s a few quick lessons from my experience:

  • Remember to be a good citizen. Scrape with a delay between each request and a unique user-agent. This may not always work, but as far as possible, make it easy for them to figure out you’re scraping.
  • ASP based websites are difficult to scrape. A bunch of Indian court websites are built on ASP and you can’t submit forms without JavaScript. I couldn’t get phantomjs or any of those libs to work either. If you can get them working, please talk to me! Sandeep has taken over from me and I’m sure he’ll find it useful.
  • Data is inconsistently inconsistent. This is a problem. You can make no assumptions about the data while scraping. The best you can do is collect everything and find patterns later. For example, a judge’s name may be written in different ways from case to case. You can normalize them later.
  • These sites aren’t highly available, so plan for retrying and backing off in your code. In fact, I’d recommend running the scraper overnight and never in the morning from 8 am to 12 pm.
  • Assume failure. Be ready for it. The first few times you write the code, you have to keep a close watch. It will fail in many different ways and you should be ready to add another Except clause to your code :)
  • Get more data than you need, because re-scraping will cost time.
  • Gujarat High Court has a JavaScript-based frontend. There’s an XHR endpoint that returns JSON. It’s the only site I’ve scraped which had a pleasant developer experience.
Categories: FLOSS Project Planets

Norbert Preining: Debian/TeX Live 2016.20160623-1

Planet Debian - Thu, 2016-06-23 22:33

About one month has passed since we did release TeX Live 2016, and more than a month since the last Debian packages, so it is high time to ship out a new checkout of upstream. Nothing spectacular new here, just lots and lots of updates since the freeze.

I am dedicating this release to those intelligent beings who voted against the stupid Brexit and for remaining in the EC! – I am still optimist!

New packages

aucklandthesis, autobreak, cquthesis, getargs, hustthesis, ietfbibs, linop, markdown, olsak-misc, optidef, sanitize-umlaut, umbclegislation, wordcount, xcntperchap.

Updated packages

academicons, achemso, acmart, acro, animate, apa6, arabluatex, archaeologie, babel-hungarian, beamertheme-epyt, beebe, biblatex-abnt, biblatex-anonymous, biblatex-bookinother, biblatex-caspervector, biblatex-chicago, biblatex-manuscripts-philology, biblatex-morenames, biblatex-opcit-booktitle, biblatex-philosophy, biblatex-realauthor, biblatex-source-division, biblatex-subseries, bidi, bookcover, bxjscls, caption, chemformula, chemmacros, circuitikz, cloze, cochineal, context, csplain, cstex, datetime2, denisbdoc, dvipdfmx-def, epstopdf, erewhon, exsol, fbb, fibeamer, fithesis, fontawesome, fontspec, fonts-tlwg, geschichtsfrkl, getmap, glossaries, glossaries-extra, graphics, graphics-cfg, gregoriotex, gzt, he-she, hook-pre-commit-pkg, hyperref, ifluatex, keyvaltable, koma-script, l3build, latex, latex-bin, limap, lollipop, lshort-chinese, luaotfload, luatex85, luatex-def, luatexja, lua-visual-debug, marginnote, mcf2graph, media9, minted, mptopdf, msu-thesis, musixtex, navigator, nwejm, oberdiek, patchcmd, pdfcomment, pdftex-def, pdfx, pkuthss, platex, pstricks, ptex, ptex2pdf, ptex-base, ptex-ng, reledmac, repere, scheme-xml, sduthesis, showlabels, tableaux, tcolorbox, tex4ht, texinfo, texlive-scripts, tex-overview, textpos, tools, translations, tudscr, unicode-data, uplatex, uptex, xassoccnt, xcharter, xetex, xindy, yathesis, ycbook.


Categories: FLOSS Project Planets

Hook 42: Block improvements in Drupal 8

Planet Drupal - Thu, 2016-06-23 20:53
Thursday, June 23, 2016

Blocks in Drupal 7 are pretty useful but, in practice, larger sites often have requirements that core blocks can't support like placing the same block in different regions for different content types. I’m happy to see core blocks have been improved in Drupal 8 to be much more practical and powerful.

I was fortunate to attend the Drupal North conference this month in Montreal. At Drupal North, Ted Bowman gave a nice presentation on the power of Drupal 8 blocks. Ted has also has created a helpful Drupal 8 contrib module called Block Visibility Groups that extends blocks even further.

In his talk, he explained some of the new abilities of blocks including:

  • Blocks can be exported
  • Blocks can be placed in different regions
  • Blocks can have fields

Let’s take a look at these new features and the Block Visibility Groups module in more depth.

Blocks can be exported

First, blocks are now exportable. In Drupal 7, block settings can be moved through the development workflow with Features, but it isn't always reliable. In Drupal 8, block settings are saved in core configuration files and these files can be checked into your favorite code repository such as git.

Example of a block configuration file in D8

Blocks can be placed in different regions

Another improvement in Drupal 8 is that a particular block can be in more than one theme region. For example, if a site needs to have the same block on two different pages in two different regions that is not possible with core blocks in Drupal 7. Site builders have to use other methods, such as using Panels or Context to achieve this.

Now, in Drupal 8, placing the same block in different regions for different pages (or even on the same page!) is possible. Let’s see how that looks.

Example of the "Help" block being placed in two regions

Blocks can have fields

Another important difference for Drupal 8 core blocks is that they can have fields. In Drupal 7, we have to use the Bean (Blocks Entities Aren't Nodes) module to have fields on our blocks.

In D8, you can add fields to blocks in the same way you add them to content types or other fieldable entities. This feature changes block versatility immensely! We can have block types of various flavors and use those in intelligent ways. For example, we could have a block type that has just links in it or one with a text blurb and an image. Using block types is much better for structured content (which is the basis of a solid content strategy).

Example of a custom block type with a custom field

Blocks are even better with contrib

The block system has improved with the help of D8 contributed modules as well. An excellent example of this is Ted Bowman’s Block Visibility Groups module. With this small module enabled, administrators can manage where all blocks are visible. This admin tool is a valuable UX improvement and an alternative to Panels for less complex sites in Drupal 8.

Have fun with your Drupal 8 blocks! Kristin Bradham - K2 Topics: Services:
Categories: FLOSS Project Planets

A. Jesse Jiryu Davis: 72% Of The People I Follow On Twitter Are Men

Planet Python - Thu, 2016-06-23 18:44

At least, that's my estimate. Twitter does not ask users their gender, so I have written a program that guesses based on their names. Among those who follow me, the ratio is even worse: 83% are men. None are gender-nonbinary as far as I can tell.

The way to fix the first ratio is not mysterious: I should notice and seek more women experts tweeting about my interests, and follow them.

The second ratio, on the other hand, I can merely influence, but I intend to improve it as well. My network on Twitter should represent of the software industry's diverse future, not its unfair present.

How Did I Measure It?

I set out to estimate the gender ratio of the people I follow—my "friends" in Twitter's jargon—and found it surprisingly hard. Twitter analytics readily shows me the converse, an estimate of my followers' gender ratio:

So, Twitter analytics divides my followers' accounts among male, female, and unknown, and tells me the ratio of the first two groups. (Gender-nonbinary folk are absent here—they're lumped in with the Twitter accounts of organizations, and those whose gender is simply unknown.) But Twitter doesn't tell me the ratio of my friends. That which is measured improves, so I searched for a service that would measure this number for me, and found FollowerWonk.

FollowerWonk guesses my friends are 71% men. Is this a good guess? For the sake of validation, I compare FollowerWonk's estimate of my followers to Twitter's estimate:

Twitter analytics menwomen Followers83% 17% FollowerWonk menwomen Followers81% 19% Friends I follow 72% 28%

My followers show up 81% male here, close to the Twitter analytics number. So far so good. If FollowerWonk and Twitter agree on the gender ratio of my followers, that suggests FollowerWonk's estimate of the people I follow (which Twitter doesn't analyze) is reasonably good. With it, I can make a habit of measuring my ratio, and improve it.

At $30 a month, however, checking my ratio with FollowerWonk is a pricey habit. I don't need all its features anyhow. Can I solve only the gender-ratio problem economically?

Since FollowerWonk's numbers seem reasonable, I tried to reproduce them. Using Python and some nice Philadelphians' Twitter API wrapper, I began downloading the profiles of all my friends and followers. I immediately found that Twitter's rate limits are miserly, so I randomly sampled only a subset of users instead.

I wrote a rudimentary program that searches for a pronoun announcement in each of my friends' profiles. For example, a profile description that includes "she/her" probably belongs to a woman, a description with "they/them" is probably nonbinary. But most don't state their pronouns: for these, the best gender-correlated information is the "name" field: for example, @gvanrossum's name field is "Guido van Rossum", and the first name "Guido" suggests that @gvanrossum is male. Where pronouns were not announced, I decided to use first names to estimate my ratio.

My script passes parts of each name to the SexMachine library to guess gender. SexMachine has predictable downfalls, like mistaking "Brooklyn Zen Center" for a woman named "Brooklyn", but its estimates are as good as FollowerWonk's and Twitter's:

 nonbinarymenwomenno gender,
unknown Friends I follow116866173   0% 72% 28%   Followers0459108433   0% 81% 19%  

(Based on all 408 friends and a sample of 1000 followers.)

Know Your Number

I want you to check your ratios, too, so I've deployed "My Ratio" to PythonAnywhere's handy service for $10 a month:


The application may rate-limit you or otherwise fail, so use it gently. The code is on GitHub. It includes a command-line tool, as well.

Who is represented in your network on Twitter? Are you speaking and listening to the same unfairly distributed group who have been talking about software for the last few decades, or does your network look like the software industry of the future? Let's know our numbers and improve them.

Image: Cyclopedia of Photography 1975.

Categories: FLOSS Project Planets

CentOS 6.8 image with Qt5.7, Python 3.5, LLVM 3.8

Planet KDE - Thu, 2016-06-23 18:05

While trying to bring my setup to package KDevelop standalone for Linux into a shape where it has a nonzero probability of me picking it up again in half a year and actually understanding how to use it, I created a docker base image which I think might be useful to other people trying to package Linux software as well. It is based on CentOS 6.8 and includes Qt 5.7 (including QtWebKit), Python 3.5 and LLVM, all built against the old CentOS libs (and thus e.g. compatible with most glibc versions out there). If you want to use it, simply install docker, and

systemctl start docker docker pull scummos/centos6.8-qt5.7 docker run -i -t scummos/centos6.8-qt5.7

Good luck with it!

If docker fails to pull the image, you might have to lower the MTU of your network device to 900 (sic).

Categories: FLOSS Project Planets

Marble Maps in KDE Randa Meetings 2016

Planet KDE - Thu, 2016-06-23 17:42

One more year of fun and intense productivity in Randa came to an end just a few days back, and I feel so good to have been a part of it. Much progress was made by the Marble team this year by Dennis, Torsten, David and me. I mostly worked on the Marble Maps Android app’s navigation feature, and would like to mention the changes here very briefly:

  • First of all, cmake was picking up the wrong target for aprs plugin, so I made cmake to skip the aprs plugin for Android, so that it does not lead to an unnecessary crash during building it.
  • There was a bug in the Route Simulation position provider plugin, it was not working in the Navigation mode. Made a fix to that.
  • Replaced the older QImage based current-position pointer, with a QML based one in the Android app. So now we are able to make our custom animations on it.
  • The current-position pointer will be staying at points on the route itself, when the user is very close to the route (not too deviated from it). With the new QML based pointer, we made sure that the radius of accuracy remains the same with respect to the Earth irrespective of which height we are viewing the map from. Plus, we have got rid of this accuracy-indicator when the position-pointer is already on the route, and are showing it only otherwise.
  • There was a minor bug with this in the desktop version as well. Imagine that while you’re simulating a navigation through a route between New York and Boston, you zoom out till the whole Earth is visible, then rotate the Earth so that the other half of the globe, for example India, is visible now, instead of America. In that case the position marker was showing up on the top-left corner of map (somewhere in the sky), when it was supposed to not be visible at all since it is theoretically present on the other “not-visible” side of the globe now. So this bug was fixed as well.
  • Used some new maneuver direction pixmaps for navigation in the Android app, getting rid of old ones, and also made them look sharper in the Navigation info-bar while they were shown during turn-by-turn navigation, by making some tweaks in the QMLs that were using them.
  • Finally, the distance text that shows on the top while in Navigation mode, its border height has been changed to match the height of that of the maneuver pixmaps shown on its left as well, so that the view looks much more uniform. Plus, a similar panel has been added at the bottom as well, that shows the speed and distance information during the navigation, each at two different sides of the panel.

That’s all about work. And adding to the yummy food that turns out to be truly delicious every single year, this time we had some more fun activities as well. We were fortunate enough to board a cable-car to ride to as high as possible in the mountains, so much that we ended up finding ourselves in snowfall ( it’s my first time being in a snowfall ), which was amazing! The special red tea we were greeted at a stop at a restaurant during the hike was pretty unique. We then visited a museum, and that was a lot of fun as well.

Such a lively and eventful week it was, thanks a lot to Mario and Simon for organizing the Randa Meetings one more year. Let’s keep the Marble rolling, and I hope to be a part of this again in the subsequent years to come!  :)

Categories: FLOSS Project Planets

Mike Driscoll: Python 101: An Intro to ftplib

Planet Python - Thu, 2016-06-23 17:30

The File Transfer Protocol (FTP) is used by many companies and organizations for sharing data. Python provides an File Transfer Protocol module in its standard library called ftplib that implements the client side of the FTP protocol. You can learn all about the File Transfer Protocol by reading the RFC 959 document on the Internet. However the full specification is outside the scope of this article. Instead we will focus on the following topics:

  • Connecting to an FTP server
  • Navigating it’s structure
  • Downloading files from the FTP server
  • Uploading files to an FTP server

Let’s get started!

Connecting to an FTP Server

The first thing we need to do is find an FTP server to connect to. There are many free ones you can use. For example, most Linux distributions have FTP mirrors that are publicly accessible. If you go to Fedora’s website (https://admin.fedoraproject.org/mirrormanager/) you will find a long list of mirrors that you can use. They aren’t just FTP though, so be sure that you choose the correct protocol or you will receive a connection error.

For this example, we will use ftp.cse.buffalo.edu. The official Python documentation uses ftp.debian.org, so feel free to try that as well. Let’s try to connect to the server now. Open up the Python interpreter in your terminal or use IDLE to follow along:

>>> from ftplib import FTP >>> ftp = FTP('ftp.cse.buffalo.edu') >>> ftp.login() '230 Guest login ok, access restrictions apply.'

Let’s break this down a bit. Here we import the **FTP** class from ftplib. Then we create an instance of the class by passing it the host that we want to connect to. Since we did not pass a username or password, Python assumes we want to login anonymously. If you happen to need to connect to the FTP server using a non-standard port, then you can do so using the connect method. Here’s how:

>>> from ftplib import FTP >>> ftp = FTP() >>> HOST = 'ftp.cse.buffalo.edu' >>> PORT = 12345 >>> ftp.connect(HOST, PORT)

This code will fail as the FTP server in this example doesn’t have port 12345 open for us. However, the idea is to convey how to connect to a port that differs from the default.

If the FTP server that you’re connecting to requires TLS security, then you will want to import the FTP_TLS class instead of the FTP class. The FTP_TLS class supports a keyfile and a certfile. If you want to secure your connection, then you will need to call prot_p to do so.

Navigating Directories with ftplib

Let’s learn how to see what’s on the FTP server and change directories! Here is some code that demonstrates the normal method of doing so:

>>> from ftplib import FTP >>> ftp = FTP() >>> ftp.login() >>> ftp.retrlines('LIST') total 28 drwxrwxrwx 2 0 0 4096 Sep 6 2015 .snapshot drwxr-xr-x 2 202019 5564 4096 Sep 6 2015 CSE421 drwxr-xr-x 2 0 0 4096 Jul 23 2008 bin drwxr-xr-x 2 0 0 4096 Mar 15 2007 etc drwxr-xr-x 6 89987 546 4096 Sep 6 2015 mirror drwxrwxr-x 7 6980 546 4096 Jul 3 2014 pub drwxr-xr-x 26 0 11 4096 Apr 29 20:31 users '226 Transfer complete.' >>> ftp.cwd('mirror') '250 CWD command successful.' >>> ftp.retrlines('LIST') total 16 drwxr-xr-x 3 89987 546 4096 Sep 6 2015 BSD drwxr-xr-x 5 89987 546 4096 Sep 6 2015 Linux drwxr-xr-x 4 89987 546 4096 Sep 6 2015 Network drwxr-xr-x 4 89987 546 4096 Sep 6 2015 X11 '226 Transfer complete.'

Here we get logged in and then we send the LIST command to the FTP server. This is done by calling our ftp object’s retrlines method. The retrlines method prints out the result of the command we called. In this example, we called LIST which retrieves a list of files and/or folders along with their respective informations and prints them out. Then we used the cwd command to change our working directory to a different folder and then re-ran the LIST command to see what was in it. You could also use your ftp object’s dir function to get a listing of the current folder.

Downloading a File via FTP

Just viewing what’s on an FTP server isn’t all that useful. You will almost always want to download a file from the server. Let’s find out how to download a single file:

>>> from ftplib import FTP >>> ftp = FTP('ftp.debian.org') >>> ftp.login() '230 Login successful.' >>> ftp.cwd('debian') '250 Directory successfully changed.' >>> out = '/home/mike/Desktop/README' >>> with open(out, 'wb') as f: ... ftp.retrbinary('RETR ' + 'README.html', f.write)

For this example, we login to the Debian Linux FTP and change to the debian folder. Then we create the name of the file we want to save to and open it in write-binary mode. Finally we use the ftp object’s retrbinary to call RETR to retrieve the file and write it to our local disk. If you’d like to download all the files, then we’ll need to a file listing.

import ftplib import os   ftp = ftplib.FTP('ftp.debian.org') ftp.login() ftp.cwd('debian') filenames = ftp.nlst()   for filename in filenames: host_file = os.path.join( '/home/mike/Desktop/ftp_test', filename) try: with open(host_file, 'wb') as local_file: ftp.retrbinary('RETR ' + filename, local_file.write) except ftplib.error_perm: pass   ftp.quit()

This example is fairly similar to the previous one. You will need to modify it to match your own preferred download location though. The first part of the code is pretty much the same, but then you will note that we call nlst which gives us a list of filenames and directories. You can give it a directory to list or just call it without and it will assume you want a listing of the current directory. Note that the nlst command doesn’t tell us how to differentiate between files and directories from its results. For this example though, we simply don’t care. This is more of a brute force script. So it will loop over the list returned and attempt to download them. If the “file” happens to actually be a directory, then we’ll end up creating an empty file on our local disk with the same name as the directory on the FTP server.

There is an MLSD command that you can call via the mlsd method, but not all FTP servers support this command. If they do, then you might be able to differentiate between the two.

Uploading Files to an FTP Server

The other major task that you do with an FTP server is upload files to it. Python can handle this too. There are actually two methods that you can use for uploading file:

  • storlines – Used for uploading text files (TXT, HTML, RST)
  • storbinary – Used for uploading binary files (PDF, XLS, etc)

Let’s look at an example of how we might do this:

import ftplib   def ftp_upload(ftp_obj, path, ftype='TXT'): """ A function for uploading files to an FTP server @param ftp_obj: The file transfer protocol object @param path: The path to the file to upload """ if ftype == 'TXT': with open(path) as fobj: ftp.storlines('STOR ' + path, fobj) else: with open(path, 'rb') as fobj: ftp.storbinary('STOR ' + path, fobj, 1024)   if __name__ == '__main__': ftp = ftplib.FTP('host, 'username', 'password') ftp.login()   path = '/path/to/something.txt' ftp_upload(ftp, path)   pdf_path = '/path/to/something.pdf' ftp_upload(ftp, pdf_path, ftype='PDF')   ftp.quit()

In this example, we create a function for uploading files. It takes an ftp object, the path of the file we want to upload and the type of the file. Then we do a quick check on the file type to determine if we should use storlines or storbinary for our upload process. Finally in our conditional statement at the bottom, we connect to the FTP server, login and upload a text file and a PDF file. An easy enhancement to add to this is some logic for changing to a specific directory once we’re logged in as we probably don’t want to just upload files to the root location.

Wrapping Up

At this point, you should know enough to get started using Python’s ftplib. It has a lot of other methods that are well worth checking out in Python’s documentation on the module. But you now know the basics of listing a directory, navigating the folder structure as well as downloading and uploading files.

Related Reading

Categories: FLOSS Project Planets

Paul Wise: DebCamp16 day 0

Planet Debian - Thu, 2016-06-23 17:16

Today is officially the first day of DebCamp 2016. The night wasn't as cold as I had feared. Woke at 5am for some reason. Noted the network still blocks port 6697 and 7000, worked around in IRC client configuration using 9999. Reply to network discussion to point that out. Minor changes to the empathy Debian RTC wiki page. Answer support@mentors.d.n bug email about shared company OpenPGP keys and suggest moving to individual keys. Review wiki RecentChanges. Comment on NetworkManager upstream bug #705545 that MAC address privacy is a complicated feature with many use cases. Warn another person that reporting Alioth to SpamCop does nothing and link to the unsubscription URL. Talk to Brown about IP address conflict sparc64 porters found with the setup of notker (sparc64 build machine). Filed Debian wishlist bug #827944 against at asking for support for using an editor to write at jobs. Woke up properly, discussed spam over breakfast. Notice Point Linux in the Distrowatch feed and invite them to the derivatives census. Point out reproducible builds in a discussion about source-only uploads. Commented that I encountered evolution upstream crash bug #680471 again. Reported gnome-shell upstream crash bug #767969. Joined the tour around the campus, enjoyed the view from the outdoor hacklab at the top of the hill. Confirmed that "Monkey Gland" from the pub menu is not in fact derived from monkeys in any way. Noted that Pollito did not eat chicken from the buffet. Beat head against VPN/SIP/WebRTC for a while but oncoming jetlag put me out of business for some hours. Pointed out the future Packages.gz removal in favour of Packages.xz to the popcon developers.

Categories: FLOSS Project Planets

Open Source and Crowdsourcing Are Not Synonyms

Open Source Initiative - Thu, 2016-06-23 16:54

OSI Board alumnus Simon Phipps recently provided some clarification to FastCo.Design around common misunderstandings related to "sourcing". We've seen more and more of these, although most often--like this example--innocent enough. However, these do provide great opportunities to remind the public about what open source actually is, and why it is so valuable.

The headline (and resulting slug) of your recent article about Mozilla unfortunately mis-states the nature of the crowdsourcing in which they are engaging by treating "open source" interchangeably with crowdsourcing. Despite sounding the same they are very different; the key difference is the ownership of the outcome.

Open source describes a pragmatic projection of the four software freedoms - to use, study, modify and distribute software for any purpose. People synchronise the fragment of their activities which relates to the software in question in a community of others with related fragmentary needs (but without a necessarily related motivation behind it). The community is of equal peers, with no one participant necessarily benefiting more than any other. True open source communities are "open-by-rule" - they have a governance that ensures no single community member can exploit the others. This approach has created the software that runs the current era of business on the web (including FastCompany which uses the open source Nginx web server to deliver pages) and has revolutionised the technology industry since the term was coined in 1998 by the founders of the Open Source Initiative.

Crowdsourcing describes the leveraging of the marginal interest and free time of a large group of people to complete a task that otherwise could not be economically completed. The result typically benefits the initiator hugely, without significantly compensating the participants. It's one of the examples of crowd behaviour James Surowiecki cites in his very interesting book The Wisdom of Crowds. Participants are crowdsourced donors, gaining no stake in the outcome in return for their effort. This is not to say I think crowdsourcing in general is a bad thing. But it's not the same thing as open source, where a community comes together for their collective mutual benefit and remain co-equal stakeholders.

- Simon Phipps

Image credit, "Crowdtesting Crowdsourcing" by Testbirds GmbH, 2005. Licensed under the Creative Commons Attribution-Share Alike 4.0 International license.
Categories: FLOSS Research

Acquia Developer Center Blog: Signal Flow, Understanding your Stack

Planet Drupal - Thu, 2016-06-23 15:20

One of the best troubleshooters we ever worked with is a former sound engineer. He taught us, “Follow the signal, it’s all signal flow.” Running a website, your stack is the combination of hardware and software that you use to deliver your website and the “signal” is a web request. To understand the “signal flow” of your site, you will need to understand your stack. Understanding how all of the pieces fit together lets you know where to start looking for the problems.

Tags: acquia drupal planetsupportdebugging
Categories: FLOSS Project Planets

PyCharm: PyCharm 2016.2 EAP 3 (build 162.1024)

Planet Python - Thu, 2016-06-23 15:12

A week ago we announced the second PyCharm 2016.2 EAP build with lots of new neat features and improvements such as:

  • vmprof Profiler support
  • An Optimize Imports configuration
  • The Thread suspend option on a breakpoint
  • Pandas Dataframes Viewer
  • and more

Today we continue the weekly delivery of PyCharm 2016.2 preview builds with the PyCharm 2016.2 EAP 3 build 162.1024. Download and try it today!

Compared to the previous EAP build, this one includes a consolidation of fixes for various bugs. The only new small change in this build is package installation from requirements.txt with extras. Now PyCharm understands the syntax of extra options in requirements.txt files, automatically installing all the specified extras. As an example, now you can use following code in your reqirements.txt files:

requests[security] ipython [notebook]

The full list of changes comparing to the previous EAP build is available in the Release notes.

Please take PyCharm 2016.2 EAP build 162.1024 for a spin! Should you encounter any problems, please report them to our public tracker.

You can download the build or use the patch-based upgrade to upgrade from within the IDE (from the previous PyCharm 2016.2 EAP build only) without a full re-installation. Just make sure you’ve selected the EAP channel in update settings.

PyCharm Team
The Drive to Develop

Categories: FLOSS Project Planets

Marcos Dione: tracing-python-execution

Planet Python - Thu, 2016-06-23 14:32

Today I stumbled upon PyCon 2016's youtube channel and started watching some of the talks. The first one I really finished watching was Ned Batchelder's "Machete debugging", a very interesting talk about 4 strange bugs and the 4 strange techniques they used to find where those bugs were produced. It's a wonderful talk, full of ideas that, if you're a mere mortal developer like me, will probably blow your mind.

One of the techniques they use for one of the bugs is to actually write a trace function. A trace function in cpython context is a function that is called in several different points of execution of Python code. For more information see sys.settrace()'s documentation.

In my case I used tracing for something that I always liked about bash: that you can ask it to print every line that's being executed (even in functions and subprocesses!). I wanted something similar for ayrton, so I sat down to figure out how this would work.

The key to all this is the function I mention up there. The API seems simple enough at first sight, but it's a little more complicated. You give this function what is called the global trace function. This function will be called with three parameters: a frame, an event and a event-dependent arg. The event I'm interested in is line, which is called for each new line of code that is executed. The complication comes because what this global trace function should return is a local trace function. This function will be called with the same parameters as the global trace function. I would really like an explanation why this is so.

The job for this function, in ayrton's case, is simple: inspect the frame, extract the filename and line number and print that. At first this seems to mean that I should read the files by myself, but luckily there's another interesting standard module: linecache to the rescue. The only 'real complication' of ayrton's use is that it would not work if the script to run was passed with the -c|--script option, but (un)luckily the execution engine already has to read the hold the script in lines, so using that as the cache instead of linecache was easy.

Finally, if you're interested in the actual code, go take a look. Just take in account that ayrton has 3 levels of tracing: à la bash (script lines prepended by +), with line numbers, and tracing any Python line execution, including any modules you might use and their dependencies. And don't forget that it also has 3 levels of debug logging into files. See ayrton --help!

ayrton python

Categories: FLOSS Project Planets

Kubuntu Dojo 2 – Kubuntu Ninjas

Planet KDE - Thu, 2016-06-23 14:22
Want to get deeper under the hood with Kubuntu?

Interested in becoming a developer?

Then come and join us in the Kubuntu Dojo:

Thursday 30th June 2016 – 18:00 UTC

Packaging is one of the primary development tasks in a large Linux distribution project. Packaging is the essential way of getting the latest and best software to the user.

We continue our Kubuntu Dojo and Ninja developers training courses. These courses are free to attend, and take place on the last Thursday of the month at 18:00 UTC.

This is course number 2, where we will look at Debian and Ubuntu packaging. Candidates will create their first packages including uploading them to their own PPA on LaunchPad, all deliver inside our online video classroom.

Details for accessing the Kubuntu Big Blue Button Conference server will be announced in the G+ event stream, and on IRC: irc://irc.freenode.net


Why it rocks

All the cool kids are doing it.
Packagers know everyone.
Not only will you be part of an elite group, but also get to know with Debian’s finest, as well as KDE developers and other application developers.

For more details about the Kubuntu Ninja’s programme see our wiki:


Categories: FLOSS Project Planets

Enthought: 5 Simple Steps to Create a Real-Time Twitter Feed in Excel using Python and PyXLL

Planet Python - Thu, 2016-06-23 14:13
PyXLL 3.0 introduced a new, simpler, way of streaming real time data to Excel from Python. Excel has had support for real time data (RTD) for a long time, but it requires a certain knowledge of COM to get it to work. With the new RTD features in PyXLL 3.0 it is now a lot […]
Categories: FLOSS Project Planets

Freelock : Ask Freelock: My host is shutting down! What should I do?

Planet Drupal - Thu, 2016-06-23 13:57

It's really a shame. Drupal Gardens has announced to its users that it's shutting down completely on August 1, and users need to move away from the service before it disappears.

It's a shame because Drupal Gardens was the only low-cost way to run a Drupal site with somebody else handling maintenance for you.

But it's not really a surprise.

DrupalDrupal PlanetSaaSHosting
Categories: FLOSS Project Planets

Aten Design Group: Making region content available to node templates in Drupal 8

Planet Drupal - Thu, 2016-06-23 13:06

Why would you need to render the content from Drupal’s block layout via a node template file? Normally, that is the territory of page templates. The use-case for me was a page where node-specific fields were mixed in with blocks to the extent that rendering region content in a page template file wasn't going to work.

I needed to be able to render my region content amidst field values in my node template files. Drupal doesn't let you do that out-of-the-box.

Superpower your Nodes

A region defined as ‘Primary Content’ (primary_content) in our theme can be printed in a page template like so:

{{ page.primary_content }}

Try that in a node template and you get a big fat nothing. Using THEME_preprocess_node we can change this, and superpower our node templates to be as capable as page templates.

Replace “THEME” with your theme name below:

/** * Implements hook_preprocess_node() for NODE document templates. */ function THEME_preprocess_node(&$variables) { // Allowed view modes $view_mode = $variables['view_mode']; // Retrieve view mode $allowed_view_modes = ['full']; // Array of allowed view modes (for performance so as to not execute on unneeded nodes)   // If view mode is in allowed view modes list, pass to THEME_add_regions_to_node() if(in_array($view_mode, $allowed_view_modes)) { // Allowed regions (for performance so as to not execute for unneeded region) $allowed_regions = ['primary_content']; THEME_add_regions_to_node($allowed_regions, $variables); } }   /** * THEME_add_regions_to_node */   function THEME_add_regions_to_node($allowed_regions, &$variables) { // Retrieve active theme $theme = \Drupal::theme()->getActiveTheme()->getName();   // Retrieve theme regions $available_regions = system_region_list($theme, 'REGIONS_ALL');   // Validate allowed regions with available regions $regions = array_intersect(array_keys($available_regions), $allowed_regions);   // For each region foreach ($regions as $key => $region) {   // Load region blocks $blocks = entity_load_multiple_by_properties('block', array('theme' => $theme, 'region' => $region));   // Sort ‘em uasort($blocks, 'Drupal\block\Entity\Block::sort');   // Capture viewable blocks and their settings to $build $build = array(); foreach ($blocks as $key => $block) { if ($block->access('view')) { $build[$key] = entity_view($block, 'block'); } }   // Add build to region $variables[$region] = $build; } }

After clearing caches, I can now print content specified in Drupal’s block layout using my node template files. For example, if I’ve specified primary_content in $allowed_regions, then I can access it via node--node_type.html.twig with:

{{ primary_content }}

Categories: FLOSS Project Planets

Drupal core announcements: Fabianx and stefan.r are now Drupal 7 co-maintainers

Planet Drupal - Thu, 2016-06-23 11:13

Earlier this year, I announced that I was seeking additional Drupal 7 co-maintainers in order to help bolster the efforts of the indefatigable David Rothstein. Thank you very much to everyone who responded to the call; there were certainly some very eager and qualified applications!

I have selected Fabian Franz (Fabianx) and Stefan Ruijsenaars (stefan.r), which are also two of the names put forward by David himself. I recently asked them to apply to become Drupal 7 co-maintainers, and I am pleased to announce they both gave an enthusiastic yes!

Fabian will be a Drupal 7 Framework Manager, and Stefan will be a Drupal 7 Release Manager and Drupal 7 Product Manager. David will continue his role as a Framework Manager, Release Manager and Product Manager for Drupal 7.

Fabian is from Germany, and is a Senior Performance Engineer and Technical Lead at Tag1 Consulting. He is always up to a challenge, and has been a part of the Drupal community for over 6 years. He was already a subsystem maintainer for the Drupal core theme system, and he has done innovative work in terms of Scalability and High Performance for Drupal (BigPipe!). He is in general passionate about everything Open Source since GNU was born, which as it happens, is the same day as his birthday.

Stefan hails from the Netherlands, and is a freelancer currently working with Belgian government clients. He has been part of the Drupal community since his first Drupalcon in Szeged in 2008, and recently became a member of the Drupal Security Team.

Both Fabian and Stefan have ample experience contributing both to numerous contributed modules, and to Drupal core, being especially instrumental to squashing the last critical bugs prior to the Drupal 8 release.

Please join me in welcoming Fabian and Stefan to the Drupal 7 core committer team! (And if you happen to be at Drupal Developer Days Milan, you can high-five Stefan in person! :-))

Categories: FLOSS Project Planets
Syndicate content