Today we announce the second early preview build of PyCharm 5.0.2. The list of bug fixes and improvements for this build can be found here.
Some highlights of the PyCharm 5.0.2 build 143.1015.4 are:
- fixes for different performance issues
- fixes for docstrings support
- a number of fixes for scientific tools, including IPython Notebook, matplotlib and miniconda support
- a number of fixes for debugger
- fixes for Docker and remote interpreters support
- a fix for Django support
- and more
Develop with Pleasure!
- Reduce friction between on premises and cloud
- Build your software platform agnostic
- The best cloud providers
- Deploying Drupal Like a Pro - Part 1: File Structure
- Drupal 8 performance: enabling the classloader cache
- Git shell on Windows reports “sh.exe has stopped working (APPCRASH)”
- Hiding the fact that your site runs Drupal
- When PHP crashes: how to collect meaningful information and what to do with it
- Making namespaced callbacks work in Drupal 7 (without hacking core and with bound parameters)
- PHP 7 nightlies for Windows
- Benchmarking Drupal 8 on PHP 7-dev
- Node Comment and Forum working together to boost user participation
- Build GIT on Windows from Sources
The #d8rules team is excited to welcome Acquia to our list of supporters. With their generous support of fully funding milestone 2, fago & klausi can plan dedicated time over the next months to focus on getting the MVP of Rules for Drupal 8 done.
Since finishing milestone 1 and DrupalCon Barcelona, we are basically in a developer preview state. The basic APIs of Rules 8 are pretty stable already, enabling contributed module porters starting work on their integrations.Milestone 2
Milestone 2 now is all about getting a useable product to developers & end users of Drupal 8.
See what's planned for M2:
Completing Rules engine features (Metadata assertions, logging service)
Rules plug-ins part two (Events, Loops, caching, components API)
Configuration management system support (config entity, CMI support, integrity checks & config schema)
Generic rules integrations (Typed data & entity support)
Entity token support
- Basically usable UI (Nothing fancy yet)
- Basic API for embedding condition and action forms
The estimated, remaining 316 hours for M2 are fully funded by Acquia, drunomics and epiqo. Acquia is putting € 14.220,- in to help us work continuously over the next months. drunomics & epiqo are providing 50 % by a lowered rate of € 45 for fago and klausi to work on #d8rules during office hours.
We are expecting a release for M2 for beginning of March, 2016. This should allow the 25 % Drupal 7 sites which use the Rules module to start building for Drupal 8. Of course, we are also looking forward to see new adapters making use of flexible, UI-driven workflows.
Thanks again for everybody helping speeding up our work of porting Rules to Drupal 8. If you'd like to help out getting funding secured for Milestone 3, let's get in contact.
This time we’ll celebrate early!
Date: Friday, 4 December 2015 (correct year this time around)
Place: Medocs Cafe am Bismarckplatz, Sofienstraße 7b, 69115 Heidelberg
Who: You! And fellow KDE developers and users
What we’re going to do: Have a few beers, a delicious dinner, talk, have fun, …
Please ping me, if you’re around and planning to come (contact info can be found in the Impressum, or tell kbroulik in #plasma on Freenode), so I can extend the reservation, if needed.
So.... Drupal 8 got released! Congrats everybody! The end of life of Drupal 6 is then final. In addition, the 16th of november it was announced that Drupal.org is now serving its content and files via Fastly; which is giving a significant performance boost, well done!
Furthermore, what I noticed last month on module updates:1) Scroll to destination anchors
This post is intended to document some elements of workflow that I’ve setup to manage videos produced for a MOOC, where different colleagues work collaboratively on a set of video sequences, in a remote way.
We are a team of several schools working on the same course, and we have an incremental process, so we need some collaboration over a quite long period of many remote authors, over a set of video sequences.
We’re probably going to review some of the videos and make changes, so we need to monitor changes, and submit versions to colleagues on remote sites so they can criticize and get later edits. We may have more that one site doing video production. Thus we need to share videos along the flow of production, editing and revision of the course contents, in a way that is manageable by power users (we’re all computer scientists, used to SVN or Git).
I’ve decided to start an experiment with Git and Git-Annex to try and manage the videos like we use to do for slides sources in LaTeX. Obviously the main issue is that videos are big files, demanding in storage space and bandwidth for transfers.
We want to keep a track of everything which is done during the production of the videos, so that we can later re-do some of the video editing, for instance if we change the graphic design elements (logos, subtitles, frame dimensions, additional effects, etc.), for instance in the case where we would improve the classes over the seasons. On the other hand, not all colleagues want to have to download a full copy of all rushes on their laptop if they just want to review on particular sequence of the course. They will only need to download the final edit MP4. Even if they’re interested in being able to fetch all the rushes, should they want to try and improve the videos.
Git-Annex brings us the ability to decouple the presence of files in directories, managed by regular Git commands, from the presence of the file contents (the big stuff), which is managed by Git-Annex.
Here’s a quick description of our setup :
- we do screen capture and video editing with Camtasia on a Windows 7 system. Camtasia (although proprietary) is quite manageable without being a video editing expert, and suits quite well our needs in terms of screen capture, green background shooting and later face insertion over slides capture, additional “motion design”-like enhancement, etc.
- the rushes captured (audio, video) are kept on that machine
- the MP4 rendering of the edits are performed on that same machine
- all these files are stored locally on that computer, but we perform regular backups, on demand, on a remote system, with rsync+SSH. We have installed git for Windows so we use bash and rsync and ssh from git’s install. SSH happens using a public key without a passphrase, to connect easily to the Linux remote, but that isn’t mandatory.
- the mirrored files appear on a Linux filesystem on another host (running Debian), where the target is actually managed with git and git-annex.
- there we handle all the files added, removed or modified with git-annex.
- we have 2 more git-annex remote repos, accessed through SSH (again using a passphrase-less public key), run by GitoLite, to which git-annex rsyncs copies of all the file contents. These repos are on different machines keeping backups in case of crashes. git-annex is setup to mandate keeping at least 2 copies of files (numcopies).
- colleagues in turn clone from either of these repos and git-annex get to download the video contents, only for files which they are interested in (for instance final edits, but not rushes), which they can then play locally on their preferred OS and video player.
Why didn’t we use git-annex on windows directly, on the Windows host which is the source of the files ?
We tried, but that didn’t make it. Git-Annex assistant somehow crashed on us, thus causing the Git history to be strange, so that became unmanageable, and more important, we need robust backups, so we can’t allow to handle something we don’t fully trust: shooting again a video is really costly (setting up again a shooting set, with lighting, cameras, and a professor who has to repeat again the speech!).
The rsync (with –delete on destination) from windows to Linux is robust. Git-Annex on Linux seems robust so far. That’s enough for now
The drawback is that we need manual intervention for starting the rsync, and also that we must make sure that the rsync target is ready to get a backup.
The target of the rsync on Linux is a git-annex clone using the default “indirect” mode, which handles the files as symlinks to the actual copies managed by git-annex inside the .git/ directory. But that ain’t suitable to be compared to the origin of the rsync mirror which are plain files on the Windows computer.
We must then do a “git-annex edit” on the whole target of the rsync mirror before the rsync, so that the files are there as regular video files. This is costly, in terms of storage, and also copying time (our repo contains around 50 Gb, and the Linux host is a rather tiny laptop).
After the rsync, all the files need to be compared to the SHA256 known to git-annex so that only modified files are taken into account in the commit. We perform a “git-annex add” on the whole files (for new files having appeared at rsync time), and then a “git-annex sync”. That takes a lot of time, since all SHA256 computations are quite long for such a set of big files (the video rushes and edited videos are in HD).
So the process needs to be the following, on the target Linux host:
- git annex add .
- git annex sync
- git annex copy . –to server1
- git annex copy . –to server2
- git annex edit .
- only then : rsync
Iterate ad lib
About 3 years ago, I wrote a piece titled "Going in the wrong direction" (well worth your time, go ahead and read it). In it, I highlighted the issue of the high cost of computing for experimentation and innovation, particularly when it comes to students. This obviously has impact on STEAM and school budgets too. I suggested that we'd see $20 and even sub $20 computers very soon.
The Raspberry Pi had established itself as a great option for exactly this. In mid 2012, during a Python conference in North Carolina (USA), I demoed a simple project using a Raspberry Pi controlling a laser. Everybody in attendance was sold on the concept of a $35 computer.
The $35, $25, I mean $20 computerWhat I imagined for the price trend
I had to revisit the original story at the beginning of 2015, because the price of each iteration of the Raspberry Pi entry level model kept going down. It looked like sub $20 was close, at least as I was picturing it in my mind. At the same time, the higher spec model kept getting better (see my article on 3D Future Tech as to why that is possible)
I'll CHIP in $9
Earlier this year, a kickstarter campaign introduced the CHIP, a $9 computer. According to http://getchip.com they will sell it this coming Monday for $8!
How low can you go?
Meet the $5 #pizero
The Raspberry Pi foundation is now selling a $5 version of the Raspberry Pi. It is half the size of the Model A+ and a quarter of the price...
And yet another price model that totally disrupts the field. Just look at that:
So now, we've reached the price level where distribution and shipping cost impact more than the cost of the computer itself. This is the next problem to solve in bridging the digital divide.
We live in interesting times...
This year I've got zero happy birthday spam message from phone, post, email, and from random people on that Internet social thing. In these days, that's a WOW, yes it is.
On the other hand, full of love and simple celebrations together with local ones. A few emails and phone calls from close friends/family who are physically distant.
I'm happier than ever with my last years' choices of caring about my privacy, not spending time with fake relationships and keeping myself an unimportant one for the $SYSTEM. That means a lot for me.
75%. This is really quite tricky!
this is really beautiful. Available as a printable, 17″ x 38″ PDF from http://www.oregongeology.org/pubs/ll/p-poster-willamette.htm
available as prints — vector art with a hint of the bacterial
This is the second year that DrupalCampMelbourne has been run in it’s current form, and it’s expected to be just as much fun as it was last year, but maybe just a bit bigger.
DrupalCampMelbourne is a two day event, with one day of sessions and one day of code sprints, but the way it’s run is a little bit unique (as far as we know). Unlike a conventional Conference or Camp, the scheduling is 100% determined by the attendees on the day.
How does that work you ask?
It’s relatively simple:
- First thing Friday morning all attendees get the opportunity to do a short lightning talk explaining the topic they wish to cover.
- During the lightning talks, all attendees will vote on the sessions they wish to see.
- Finally, after all lightning talks and voting is complete, the DrupalCampMelbourne website auto-magically builds the schedule based off the votes, number of sessions and room sizes.
We ran this approach for the first time last year and it worked superbly, and with a little tweaking to the algorithm this year we expect it to be just as good, if not better.
The major benefits of this approach are:
- Everyone gets an opportunity to have their say, both in submitting a session and in voting on what they’d like to see.
- No “committee” or “track chairs” are required to vet every talk and make the final decisions, reducing the organisational time of the event.
- SkyNet is one step closer to taking hold of us all… oh wait.
So if you are coming (you are coming right?), make sure to get their early and have your say. And remember, everyone has something worth saying and worth hearing, and there’s nowhere better to start than a local community.
There are still some tickets left for the event, so if you haven’t got yours, get it now: http://dcm2015.drupal.org.au
The future (a.k.a, SkyNet?)
The auto-magic scheduling of the talks is but the beginning, just as the day of session is just the beginning of DrupalCampMelbourne.
Day 2 of DrupalCampMelbourne is, as it was last year, a Code Sprint. This year, I will be running a sprint on the future of the DrupalCampMelbourne website in the hopes to make it even better; more autonomous, more usable and also more generic.
The “auto-magic” scheduling feature is a great help for running a DrupalCamp, it helps get Day 1 all sorted with minimal effort, but it’s not the only part that can be automated and improved. A larger portion of the camp itself could be automated.
If, when setting up the next camp, one where to provide the site with the date of the event, the camp could set a schedule for the organisers (when to have venue booked by, when to contact sponsors, etc), it could transition through various states (register your interest, event information, signup, etc), it could manage the budget (venue cost + resource costs - sponsors - tickets = success) and much more.
The possibilities are endless.
There’s no question that there have been some issues, certain information lacking, not enough communication, and other various management related issues; this is inevitable when the number of volunteers is in the low single digits and the time those volunteers have is equally lacking.
DrupalCampMelbourne is as open source as it can be at the moment, the source code is entirely available on GitHub.com (http://github.com/drupalmel/drupalcampmel) for anyone at all to contribute to. This year I want to push forward and get more people involved, let’s ensure the site is more usable in the future, makes more information available and the site provides the missing communication it needs.
There is absolutely no reason that this project should be specific to DrupalCampMelbourne, nor even a DrupalCamp at all, it could apply to any type of Camp style event in any locale.
Genericising the existing work and building a new DrupalCamp/Camp Drupal distribution has been a goal from the very start, and with Drupal8 out it’s the best time to do exactly that.
So come along to DrupalCampMelbourne 2015 on Saturday (and Friday) and get involved. This is only one of the various sprints that will be being run during the Code Sprint. And don’t forget, a Code Sprint isn’t just for developers, there’s something for everyone, from novice to professional.drupal planetdrupalcampmelbournedrupal
As with many crazy ideas this all started in a bar. No ordinary bar, this was DrupalCon Barcelona Trivia Night. @BarisW said “so what are you going to do about Drupal 8’s release, we’re relying on you”. I was phased, but it set my mind in motion.
I am one of those people who, when faced with a challenge, believe the impossible is possible. How do I reach every corner of the globe and spread the word of this amazing new Drupal 8 thing? This is the story of how Celebr8D8 came to be, and I didn't do it alone.
Photo: @pvishnuvijayan Bangalore Drupal 8 Release Party.Why was so driven to this?
Simple. Whilst I have been very fortunate to have attended many DrupalCons, experienced the scale and diversity of the Drupal community first hand, many have not. Many never will. Few have any concept of the global movement which builds the software. It all happens behind closed doors. I wanted to show to the world there are people in every place across the globe involved in Drupal. Of every creed, colour and background. I wanted to to transcend geography, make everyone feel united.Germinating the ideas
So I decided it would be a interesting project to try and persuade as many people as I could muster, from as broad backgrounds as I could get, to film themselves saying a script about Drupal 8. This could then be edited to feature each person in sequence. The inspiration came from these two pop videos. Cry and Band Aid.
Like most big ideas in Drupal this was going to need help. Enter Jeffrey A. "jam" McGuire, Robert Douglass and Campbell Vertesi. A few emails later we had a concept. Then out the blue Campbell suggested Celebr8D8. BOOM! I instantly knew this was the stroke of genius upon which to hand our idea from. Jam and I had skype chats, created obligatory google docs, sent dozens of persuasive emails. In less than 2 days between us we had a concept, script and list of 20+ people willing to take part.
Then out the blue Campbell suggested Celebr8D8. BOOM! I instantly knew this was the stroke of genius
But then I thought, wouldn’t it be amazing to make a Drupal 8 site for this film to live on. And how about we secretly approach dozens of people to film themselves talking about what Drupal 8 meant to them. Using Jam’s and my little black books we emailed a large number of Drupal friends and contacts sample videos inviting people to join in.
Meanwhile I twisted a few arms - Amy Leak (Designer), Matt Smith (Developer), Alison Hover (Themer), James Hall (Site Builder) all committed to creating the Drupal 8 site. They were amazing, literally a self managed team who made magic.
It was all coming together, or so I thought. Time was passing and I was becoming anxious. As social media lead for Drupal, I was one of the few people who knew down to the hour when Drupal 8 was coming out. 4 days to go and no crowd sourced videos. I stayed up until the early hours emailing people, persuading them to commit. Slowly the films started to trickle in, then a wave.
Meanwhile, the feature film submissions were coming in thick and fast. Enter Graham Brown (@vaccineMedia), producer of the film. Working into the early hours for several days with little direction he created the film we all saw watched on release day. And then Jam told us he’d had a call to go to Antwerp to meet Dries, with a film crew (thanks Acquia!). Suddenly we an exclusive, the big guy was in. Sometimes I think destiny has a part to play in life, this was one of those days.
And then Jam told us he’d had a call to go to Antwerp to meet Dries, with a film crew (thanks Acquia!). Suddenly we an exclusive, the big guy was inBut how do we get people to know about this site?
So we had an amazing film coming together, a site and dozens of community films flying in from around the globe. But how do we draw an audience? Well if there’s one I’ve learnt in Drupal, if you can inspire the community into action amazing things happen. So I set about enabling another mad idea I had - “The social media Mexican wave”.
I thought that if I could contact a person in every country where there is Drupal and ask them to retweet one tweet about our site, at a certain time in their local timezone, we could achieve something interesting. Chasing the sun around the globe, a steady rhythm of retweets would ripple across the world like the Mexican Waves in 1990’s football crowds.
So I created a webform for people to volunteer, sent a couple of tweets linking to the form and went to bed. In the morning I had a tonne of retweets and more importantly 199 volunteers with a potential reach of 350000 people! Wow! So I emailed the volunteers and primed them with my plan so that come Drupal 8 release day they could support us on social media, give us a boost!
Another secret piece to my social media master plan was to ask people's Twitter username when they submitted films. Whilst moderating each film, I took a screenshot, added it to a scheduled tweet in Hootsuite with a link to the page on our site, and @mentioned the person in the tweet. Doing so would guarantee that person noticed, hopefully was flattered and duly retweeted. Each tweet was carefully primed to go out at the right time, when that person was awake, in their part of the world. Naturally most people retweeted and boosted our impact. (an export of these tweets is available to download below, with stats so you can copy my ideas)
The hashtag #Celebr8D8 was the icing on the cake. The Drupal Association contacted me and we asking what hashtag could be used for the 207 release parties happening around the world. They loved the one we had planned and asked if they could use it too. Well no one owns hashtags, I thought it would be fun and more effective to combine efforts.
So come 9am 19th November 2015 the first tweet announcing the site went out as I slept. Thanks Hootsuite (and many others to follow that day). In New Zealand Josh Waihi’s short film launched the site with a humble tweet, and lit the touch paper of what became a 24 hr period which saw our campaign reach 250,000+ people. The Drupal community really got behind the idea, our films were watched thousands of times, a huge feeling of being connected was achieved.
We were very fortunate to have the full power of Platform.sh, provided for free (thanks Robert Douglass!), so we knew that no matter how busy the site got, it would stay up. I’m sure Platform hardly noticed, but I was pretty stoked when I noticed there were 386 visitors on the site at one moment.Was it a success?
What started as an idea by one person and a few friends took flight and it felt like the whole world joined us. People from 115 countries in the world came to the site, watched our films. For the 7 hours between 10:00 and 17:00 GMT the site sustained over 200 concurrent users. We supported 7,396 sessions by 5,474 users and 22,061 page views. Not bad for an entirely volunteer team who had less than 2 weeks notice.
222.5K impressions in 3 days. The main announcement Tweet reached 48,042 people with 309 retweets.
— Celebrate Drupal 8 (@celebr8d8) November 18, 2015
Celebr8 Drupal 8 the Film
Graham's headline film was watched 1596 times in 24 hours.Who's were the most popular films
I know how the Drupal community are so competitive, so these are in popularity order.
- Dries and Jam Belgium and NZ
- MortendK Denmark
- Drupal Association USA
- Grienauer Austria
- Noah Australia
- Net Studio Greece
- Lewis and Emma UK
- Andrew McPhaerson UK
- Amazee Switzerland
- Steve Purkiss UK
- Dave Hall Australia
You will see below more remarkable stats I’ve taken from Google Analytics, Twitter Analytics and Bitly. In the spirit of open source, there is also a spreadsheet with some of the top level stats and some PDF’s you may use and distribute freely under Creative Commons Attribution-ShareAlike 3.0 license https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribu...A huge thank you too ....
All of this makes me very happy. But it would not have been possible without some very special people. I’d like to close by saying a huge thanks to Jeffrey A. "jam" McGuire who spent for best part of two weeks helping me pull my mad plan together. Without Jam this would not have been possible, nor as amazing. And to Graham, Matt, Ali, James and Amy for tirelessly working on the film and site!
Let’s do it again some time, but for now, can I have a rest?
In the meantime here are few of my favourite tweets from this very memorable day .....Following are some of my favourite tweets
With so many to choose from, here are a selection from the tweets I favourited on Twitter.Drupal Saudi Arabia
— Essam Al-Qaie (@EssamAlQaie) November 19, 2015
— Maria Blum (@BlumCodes) November 19, 2015
— Amazee Labs Austin (@amazeelabs_atx) November 12, 2015
— Duo (@DuoConsulting) November 19, 2015
— Prafful Nagwani (@nagwaniz) November 19, 2015Drupal Nigeria
— Aniebiet Udoh (@almaudoh) November 23, 2015Drupal Bangalore
— pvishnuvijayan (@pvishnuvijayan) November 21, 2015
— AddWebSolution (@AddWebSolution) November 21, 2015
— OpiumWorks (@OpiumWorks) November 20, 2015
— cafuego (@cafuego) November 20, 2015
— Manuel Alejandro Gil (@manuelgil) November 19, 2015
— Holly Ross (@drupalhross) November 20, 2015
— Drupal Association (@DrupalAssoc) November 19, 2015
— Blisstering (@Blisstering) November 19, 2015
— Amazee Labs Austin (@amazeelabs_atx) November 19, 2015
— Isabell Schulz (@murgeys) November 19, 2015File: tweet_activity_metrics_celebr8d8_20151029_20151126_en.csv Analytics All Web Site Data Location 20151117-20151124-2.pdf Analytics All Web Site Data Location by City.pdf
Over 1500 cities
Over 1500 citiesFurther information: Campaign and site visitor stats - Google Doc
Recently the FSF's application to Let's Encrypt's Limited Beta program was accepted. For those of you who have not been following, the project Let's Encrypt is a non-profit Certificate Authority (CA) run by the Internet Security Research Group (ISRG). The IRSG, founded in 2013, is comprised of board members from various places, including but not limited to: The Electronic Frontier Foundation (EFF), and Mozilla. The main drive behind Let's Encrypt is to make the process of getting X.509 certificates for Transport Layer Security (TLS) encryption a trivial process, as well as cost-free. In addition, Let's Encrypt aims to make all this available using only Free as in Freedom software for both their server and client infrastructure. By doing this, the Let's Encrypt project hopes to make HTTPS (encrypted web traffic) the default state of the entire Internet. For further reading on the goals and mission statement of the Let's Encrypt CA, check out https://letsencrypt.org/about.
As part of the the Limited Beta program we have been granted the ability to generate certificates. This is done using the Let's Encrypt client software, which uses their API to generate and sign certificates for several of our most used domains. Using the Let's Encrypt client software makes deploying certificates almost effortless. The current procedure is fairly straight forward: Clone the git repository, run the client, allow it to grab packages it requires, and then step though a fairly pretty curses interface that takes your email address and the domain of the certificate you are trying to generate and have signed. This process, however, does require minimal downtime as the Let's Encrypt access requires the ports the web server uses to complete the API transaction. In practice this only took a minute or so of downtime to complete.
As part of our participation in the Limited Beta program, an example of the Let's Encrypt CA service can already be found on our network. Currently, we have migrated https://libreplanet.org to Let's Encrypt. The fingerprints for our new certificate, signed by Let's Encrypt, are as follows:
- SHA1: D0:48:06:70:21:7B:70:09:D0:5C:17:22:B1:C8:E9:40:EA:BA:AF:29
- SHA-256: 6B:0A:B6:4A:1A:7D:30:5C:1D:3C:A7:12:95:06:DC:F1:AA:48:7B:33:C2:81:C8:46:4E:97:1D:91:18:74:3A:98
Overall we are very excited to participate in the Let's Encrypt project. If generating X.509 certificates can be so effortless, in my opinion the end goal of a World Wide Web that defaults to HTTPS instead of HTTP is achievable. The public beta for Let's Encrypt is scheduled to go live December 3rd, 2015. I encourage any reader who runs a web server to give the Let's Encrypt CA a serious look.
Last month I wrote about context managers and a company called Webucator asked me for my permission to turn the article into a screencast. It ended up looking pretty good. Check it out below:
Webucator has other Python related training too. I don’t know much about them, but if this video is any indication, I think they would be worth checking out.
[me] Sure. What module are you using?
[friend] BeautifulSoup, but man, this is hard. It's this url...
[me] Wait, this is not a Coursera assignment you are asking me to do, is it?
[friend] No, no. I saw this thing using a different programming language and I want to do it in Python.
[me] Ok, sounds reasonable.
The URLThe basic URL that documents James Bond movies on wikipedia is at: https://en.wikipedia.org/wiki/List_of_James_Bond_films but the URL he sent me was: https://en.wikipedia.org/w/index.php?title=List_of_James_Bond_films&oldid=688916363 and hence why it looked like a assignment.
Let me pause for a brief second on this subject. I'm a big fan of reproducible research, and selecting a specific revision of a document is an excellent idea. This page will never change, whereas any given normal URL on wikipedia changes all the time.
I'll have some of that BeautifulSoupMy friend mentioned he was trying to use BeautifulSoup but facing some challenges. BeautifulSoup and lxml are the usual suspects when it comes to doing web scraping (and using requests itself to pull the data in). But I have to admit, most of the time I don't use any of these. You see, I'm lazy, and typically these solutions require too much work. If you want to see what I'm talking about, you can check using-python-beautifulsoup-to-scrape-a-wikipedia-table
I don't like to type more code than I need to. At any rate, the goal was to get the web page, parse two tables and then load the data in a pandas data frame to do further analysis, plots etc.
Enter the PandasAnd it's not even the Kung Fu Panda, just good old Pandas, the data wrangling tool par excellence (https://pypi.python.org/pypi/pandas/0.17.1).
Everybody knows, I hope, that it has a superb support for loading excel and CSV files. It's why Python is the number 1 data wrangling programming language.
But what about loading tables from wikipedia web pages, surely there is nothing that can simplify this, is there? If you've attended all PYPTUG meetings, you already know the answer.
import pandas as pd
wiki_df = pd.read_html("https://en.wikipedia.org/w/index.php?title=List_of_James_Bond_films&oldid=688916363", header=0)
read_html returns a list of dataframes, with each table found on the web page being a dataframe. So to access the box office table on this page, we have to look at the second dataframe, the first being the warning table at the top of the page. Since it is 0 indexed we refer to it with wiki_df. We don't want line 0 because that's sub headers, and we don't want the last two lines because one is a movie that's just been released and the numbers are not in yet, and the other one because it's a total column. How do we do this? Good old Python slices:
df = wiki_df[1:24]
And that's it, seriously. One line to ingest, one line to cleanup.
The result Title Year Bond actor Director Box office Budget Salary of Bond actor Box office.1 Budget.1 Salary of Bond actor.1 1 Dr. No 1962 Connery, SeanSean Connery Young, TerenceTerence Young 59.5 1.1 0.1 448.8 7.0 0.6 2 From Russia with Love 1963 Connery, SeanSean Connery Young, TerenceTerence Young 78.9 2.0 0.3 543.8 12.6 1.6 3 Goldfinger 1964 Connery, SeanSean Connery Hamilton, GuyGuy Hamilton 124.9 3.0 0.5 820.4 18.6 3.2 4 Thunderball 1965 Connery, SeanSean Connery Young, TerenceTerence Young 141.2 6.8 0.8 848.1 41.9 4.7 5 You Only Live Twice 1967 Connery, SeanSean Connery Gilbert, LewisLewis Gilbert 101.0 10.3 0.8 + 25% net merch royalty 514.2 59.9 4.4 excluding profit participation 6 On Her Majesty's Secret Service 1969 Lazenby, GeorgeGeorge Lazenby Hunt, Peter R.Peter R. Hunt 64.6 7.0 0.1 291.5 37.3 0.6 7 Diamonds Are Forever 1971 Connery, SeanSean Connery Hamilton, GuyGuy Hamilton 116.0 7.2 1.2 + 12.5% of gross (14.5) 442.5 34.7 5.8 excluding profit participation 8 Live and Let Die 1973 Moore, RogerRoger Moore Hamilton, GuyGuy Hamilton 126.4 7.0 n/a 460.3 30.8 n/a 9 man with !The Man with the Golden Gun 1974 Moore, RogerRoger Moore Hamilton, GuyGuy Hamilton 98.5 7.0 n/a 334.0 27.7 n/a 10 spy who !The Spy Who Loved Me 1977 Moore, RogerRoger Moore Gilbert, LewisLewis Gilbert 185.4 14.0 n/a 533.0 45.1 n/a 11 Moonraker 1979 Moore, RogerRoger Moore Gilbert, LewisLewis Gilbert 210.3 34.0 n/a 535.0 91.5 n/a 12 For Your Eyes Only 1981 Moore, RogerRoger Moore Glen, JohnJohn Glen 194.9 28.0 n/a 449.4 60.2 n/a 13 Octopussy 1983 Moore, RogerRoger Moore Glen, JohnJohn Glen 183.7 27.5 4.0 373.8 53.9 7.8 14 view !A View to a Kill 1985 Moore, RogerRoger Moore Glen, JohnJohn Glen 152.4 30.0 5.0 275.2 54.5 9.1 15 living !The Living Daylights 1987 Dalton, TimothyTimothy Dalton Glen, JohnJohn Glen 191.2 40.0 3.0 313.5 68.8 5.2 16 Licence to Kill 1989 Dalton, TimothyTimothy Dalton Glen, JohnJohn Glen 156.2 36.0 5.0 250.9 56.7 7.9 17 GoldenEye 1995 Brosnan, PiercePierce Brosnan Campbell, MartinMartin Campbell 351.9 60.0 4.0 518.5 76.9 5.1 18 Tomorrow Never Dies 1997 Brosnan, PiercePierce Brosnan Spottiswoode, RogerRoger Spottiswoode 338.9 110.0 8.2 463.2 133.9 10.0 19 world !The World Is Not Enough 1999 Brosnan, PiercePierce Brosnan Apted, MichaelMichael Apted 361.8 135.0 12.4 439.5 158.3 13.5 20 Die Another Day 2002 Brosnan, PiercePierce Brosnan Tamahori, LeeLee Tamahori 431.9 142.0 16.5 465.4 154.2 17.9 21 Casino Royale 2006 Craig, DanielDaniel Craig Campbell, MartinMartin Campbell 594.2 150.0 3.4 581.5 145.3 3.3 22 Quantum of Solace 2008 Craig, DanielDaniel Craig Forster, MarcMarc Forster 576.0 200.0 8.9 514.2 181.4 8.1 23 Skyfall 2012 Craig, DanielDaniel Craig Mendes, SamSam Mendes 1108.6 150.0—200.0 17.0 879.8 158.1 13.5
Marble, the swiss army knife for maps and globes, developed in the KDE community, this year has got an app variant added that is concentrating on maps and that is designed for today’s typical mobile devices with touch UI, called Marble Maps. It currently is in Beta state. Read “Announcing Marble Maps for Android Open Beta” for more info.
At this year’s Randa meeting for a few days I joined the people working on having KDE software run on Android. After all there are gazillions of such systems out there, and it only makes sense to have existing FLOS software, such as KDE ones, also run there, as a matter of dealing with realities. With the developers of Marble, KAlgebra and GCompris around, some experience could be shared, and I compiled what I learned into a small Hello KWorld example.
As a Marble fanboy, Marble Maps was my obvious toy project here when learning more about building Qt5-based software for Android. I had no Android device available, but a Jolla phone with SailfishOS. That one also has support for running Android apps, so I still could use it to test my self-built Marble Maps Android packages.
Now, it felt strange to run a Qt5-based application via Android emulation layer on an OS like SailfishOS which itself is using Qt5 and then some usual “Linux” software stack/middleware. Should it not simply run directly there?
So last week finally I gave it a try to build and to run Marble Maps natively on SailfishOS. And was presented with a problem: SailfishOS in the latest version is (still) at Qt 5.2 (with QML upgraded to 5.3 it seems) and, more important, has no QtQuick Controls. Which are used in Marble Maps for all controls. At least all the Marble C++ code was building fine after 2 small fixes, so that part was looking promising.
An option might have been to build a custom version of QtQuick Controls. But that somehow smelled like problems waiting. And I also was tempted to try to do some native UX to gather more experience with QtQuick and the SailfishOS ideas. So I forked the Marble Maps code and started to write a SailfishOS variant, using its Silica UI components. The code for Marble Maps is mainly the UI layer one, written in QML, with the actual business logic nicely in the shared Marble libs and components, so it is not that much logic which is duplicated here. Still it is not “Write once and run everywhere”. It’s up for discussion how much native multi-platform apps should go. For now I will play more with it.
Right now Marble Maps for SailfishOS almost has all current features of the “normal” Marble Maps implemented, modulo some small bugs:
These days sadly many people hearing “marble” and “sailfishos” might rather think of “tombstone”, bad timing here. But until that OS is pushing up the water lilies I will play more with it, after all I can use the code on a working device of mine.
If you want to use the code on a device of yours, find first RPMs on build.merproject.org and the code in branch “sfos” in my Marble repo clone clones/marble/kossebau/marble on the KDE git servers, see src/apps/marble-maps-sailfishos/ for the app code. Once it’s stable enough the code will be proposed for inclusion in the official Marble repo.
(Update: when installing the RPM on the Jolla phone, make sure you have “qt5-qtscript” installed, pkcon install qt5-qtscript in the terminal should get you there.)
Also looking forward to see the “normal” Marble Maps and other Marble-based apps soon on Plasma Mobile hopefully.
And anyone looking into an Ubuntu Touch version? Would that work with stock Marble Maps, thanks to QtQuick Controls, or would more native UX be needed there as well?
When load testing, we simulate user interaction with a website, increase the frequency or the number of interactions and collect the results of system usage, then analyze them to aid system improvement towards desired results. The data will prove useful for creating benchmarks of site performance, which can be compared with earlier site's performance if a site is undergoing a migration.
Grzegorz Śliwiński: Creating a timestamp in python - difference between calendar.timegm and time.mktime
There are two things people are probably not aware. First one can be attributed due to the fact that their docs are not being read properly, or more often skipped is that time.mktime function treats timetuple as if it were representing date and time in machine local time. This means even if you extract timetuple from your datetime object using utctimetuple method resulting timestamp will still be a given date in your system local time zone. Might not be a problem at all on production, but will be a much bigger issue on your machine, when you pull some data to analyse. calendar.timegm on the other hand, treats all input data as if it were passed in utc.
Speaking of which, none of those two functions accepts a parameter that would tell which timezone the timetuple is in. The only difference is that time.mktime returns a float and calendar.timegm returns an integer.
Second things - timegm function is faster than mktime! See the short benchmarks below:
Read more… (1 min remaining to read)
I’m a dummy. I pulled the old “pretend to ask an honest question but really to confirm your belief” trick and, shockingly-not, got refuted. My small sample seems to like making their Python virtual environments outside of their projects.Background
I do a lot of Python explaining (Py-splaining?) and, contrary to the “see how much our Won-The-War framework can do in minus ten lines of hello”, I try to suck it up and show the right way. This usually means, make a virtual environment. It also means make a package, which ranks up there with “drop a frozen turkey in a deep fryer using your teeth” on the scale of holiday fun.
Virtual environments are a little squirrelly to explain. Like “mobile first” for web dev, I have committed to Py3-first for teaching, which means you get pyvenv for free. I then show making a setup.py and doing pip install -e . (or python ./setup.py develop), binding that virtualenv to this project.
In my mind, that means the virtualenv is part of this project directory. If I give up on the project, I delete the directory, and everything goes with it. If I’m using SQLite, I store my .sqlite file in that working directory as well. If I’m doing JS frontend stuff, my node_modules is in that directory. It thus makes sense that my virtualenv (or plural, if I’m doing multiple Python versions) should be in the same working directory. It’s an artifact of that project.
In fact, Python’s “Hitchhiker’s Guide to Python” advertises this workflow.
I posed the question on Twitter because PyCharm, when making a new project, wants the virtualenv outside of the to-be-created project directory. I was looking to bolster my viewpoint.The Herd Is Wrong
The wisdom of crowds voted against me 12-6. That’s the trouble with crowds, they are only wise when they agree with me. I guess that’s the difference between a crowd and a mob.
I think I’ll march on, teaching to firmly associate the virtualenv with the project, holding up the Hitchhiker’s Guide as my certificate of authenticity. But I won’t file a likely-disruptive ticket in PyCharm.
There are times when working on data science problems with your local machine just doesn’t cut it anymore. Maybe your computer is old, and can’t work with larger datasets. Or maybe you want to be able to access your work from anywhere, and collaborate with others. Or maybe you have an analysis that will take a long time to run, and you don’t want to tie up your own computer. In these cases, it is useful to run Jupyter on a server, so you can access it through a browser.
Visit our site to listen to past episodes, support the show, and sign up for our mailing list.Summary
The Software and Data Carpentry organizations have a mission of making it easier for scientists and data analysts in academia to replicate and review each others work. In order to achieve this goal they conduct training and workshops that teach modern best practices in software and data engineering, including version control and proper data management. In this episode we had the opportunity to speak with Maneesha Sane, the program coordinator for both organizations, so that we could learn more about how these projects are related and how they approach their mission.Brief Introduction
- Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
- Subscribe on iTunes, Stitcher, TuneIn or RSS
- Follow us on Twitter or Google+
- Give us feedback! Leave a review on iTunes, Tweet to us, send us an email or leave us a message on Google+
- I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com
- This episode is sponsored by Zato - Microservices, ESB, SOA, REST, API, and Cloud Integrations in Python. Visit zato.io to learn more about how to integrate smarter in the modern world.
- I would also like to thank Hired, a job marketplace for developers, for sponsoring this episode of Podcast.__init__. Use the link hired.com/podcastinit to double your signing bonus.
- Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $10 credit to try out their fast and reliable Linux virtual servers for your next project
- We are recording today on November 10th, 2015 and your hosts as usual are Tobias Macey and Chris Patti
- Today we are interviewing Maneesha Sane about Software Carpentry and Data Carpentry
Zato is a modern, Python powered platform for integrating all of your data. Check it out today and tell them thank you for being a sponsor of Podcast.__init__ - zato.io
On Hired software engineers & designers can get 5+ interview requests in a week and each offer has salary and equity upfront. With full time and contract opportunities available, users can view the offers and accept or reject them before talking to any company. Work with over 2,500 companies from startups to large public companies hailing from 12 major tech hubs in North America and Europe. Hired is totally free for users and If you get a job you’ll get a $2,000 “thank you” bonus. If you use our special link to signup, then that bonus will double to $4,000 when you accept a job. If you’re not looking for a job but know someone who is, you can refer them to Hired and get a $1,337 bonus when they accept a job.
Interview with Maneesha Sane
- How did you get introduced to Python?
- Can you explain what the Software and Data Carpentry organizations are and what their respective goals are?
- What is the history of these organizations and how are they related?
- What does a typical Software Carpentry or Data Carpentry workshop look like?
- What is the background of your instructors?
- Can you explain why Python was chosen as the language for your workshops and why it is such a good language to use for teaching proper software engineering practices to scientists?
- In what ways do the lessons taught by both groups differ and what parts are common between the two organizations?
- What are some of the most important tools and lessons that you teach to scientists in academia?
- Do you tend to focus mostly on procedural development or do you also teach object oriented programming in Software Carpentry?
- What is the target audience for Data Carpentry and what are some of the most important lessons and tools taught to them?
- Do you teach any particular method of pre-coding design like flowcharting, pseudocode, or top down decomposition in software carpentry?
- What scientific domains are most commonly represented among your workshop participants for Software Carpentry?
- What are some specific things the Python community and the Python core team could do to make it easier to adopt for your students?
- What are the most common concepts students have trouble with in software & data carpentry?
- How can our audience help support the goals of these organizations?