FLOSS Project Planets

The How part (2) - Adding styling for various geographical features

Planet KDE - Sun, 2016-08-21 08:00

( This post is related to my GSoC project with Marble. For more more information you can read the introductory post )

As I mentioned in my earlier post, I modified the polyshp2osm tool so as to support conversion of Natural Earth shapefiles to OSM format so that Marble is able to render geographical features(roads, rivers, reefs, airports) provided by the Natural Earth data set. Now the primary problem was that Marble never supported the rendering of many of the geographical features provided by Natural Earth. In order to enable rendering of a new geographic feature(reef, boundaries), one needs to describe how the feature must be styled by describing the colors to be used, the width of the boundaries, any geometric patterns or any icon which needs to be associated with the geometry.

Before describing my work let me first describe how Marble handles the styling for the vector OSM theme. Marble reads the OSM file and for any geographical feature (roads, rivers, reefs, airports, shops, buildings etc) assigns it a GeoDataVisualCategory value. The geometry associated (point, polygon, linestring ) with the geographical feature is primarily styled according to this GeoDataVisualCategory. Depending on the presence of other OSM key=value pair tags and the current zoom level of the map, Marble might dynamically change the styling.

In fact, this ability to dynamically change the styling was recently added by my mentor Dennis Nienhüser during the Randa meetings.

In most of the cases, for adding the styling for a new geographic feature in Marble, one needs to do the following steps

  • Create a new GeoDataVisualCategory
  • Assign a default style to this GeoDataVisualCategory by describing the colors, width, pattern, icon which are associated with the feature
  • Decide the order of this new visual category w.r.t other visual categories. Visual categories which have a higher priority will be drawn on top of visual categories which have a lower priority.
  • Decide the minimum zoom level at which this new visual category starts appearing.
  • Map this visual category to some corresponding OSM key-value pair.
  • Decide if you want to change the default styling associated with this GeoDataVisualCategory on some particular zoom level or when some particular OSM key=value tags are encountered.
Natural Reefs

Has an OSM equivalent natural=reef which I used for tag mapping it to the styling of reefs.

Landmass

Natural Earth provides the data for countries, continents, urban areas, islands as polygons. In order to provide a common tag mapping for these land polygons, I constructed a custom marble_land=landmass for tag mapping the above land polygons to the styling of rendered land areas in Marble.

Administrative Boundaries

The OSM administrative boundaries have an admin_level=x tag which defines the size/importance of a particular administrative boundary. I tag mapped these admin_level=(1..11) tags to 11 different GeoDataVisualCategories .

Salt Lakes

Instead of creating a new visual category, I added an additional OSM tag salt=yes to the list of key=value tags list of salt lakes while keeping the visual category as NaturalWater. Now when a salt lake in encountered during parsing, Marble assigns it the default styling of NaturalWater which is a shade of blue with blue outline. However due to the tag salt=yes the styling changes to a shade of yellow with a dashed blue line.

Antarctic Ice Shelves

I created a new visual category NaturalIceShelf for styling the antarctic ice shelves and tag mapped it to natural=glacier and glacier:type=shelf

Marine Indicators

I tag mapped the OSM tag maritime=yes to a new visual category BoundaryMaritime whereas used the tags (boundary=administrative, admin_level=2, maritime=yes, border_type=territorial) for dynamically changing the styling of disputed marine boundaries.

International Date Line

Created a custom OSM tag marble_line=date and tag mapped it to a new visual category InternationalDateLine.

Bathymetries

Styling Bathymetries was quite tricky since I was allowed to use only a single visual category for styling different kinds of bathymetries. Initially I thought that this was going to be straightforward and I only need to pass the elevation info of the bathymetry and depending on this info, I can then dynamically change the styling. However I eventually found that in order to do the above task, I need go down a layer of abstraction and hack a few things at the graphics level. The problem was that the bathymetries at level 200 and 4000 were getting mixed up. Ideally bathymetries at level 4000 should appear above the bathymetry at level 200 but since the z-value(which decides which feature must be rendered on top of others) is fixed for a particular GeoDataVisualCategory, rendering the bathymetries was getting messed up. I had to make special cases in the graphics related methods of Marble so as to circumvent this problem.

Categories: FLOSS Project Planets

Creating a Photo-Box with the help of KWin

Planet KDE - Sun, 2016-08-21 07:41

For a family celebration I wanted to create a “Photo-Box” or “Selfie-Box”: a place where the guests can trigger a photo of themselves without having to use tools like a selfie-stick.

The requirements for the setup were:

  • Trigger should be remote controlled
  • The remote control should not be visible or at max hardly visible
  • The guests should see themselves before taking the photo
  • All already taken photos should be presented in a slide show to the guests

The camera in question supported some but not all of the requirements. Especially the last two were tricky. While it supported showing a slide show of all taken photos, the slide show ended as soon as a new photo was taken. But the camera also has an usb-connector so the whole power of a computer could be taken in.

A short investigation showed that gphoto2 could be the solution. It allows to completely remote control the camera and download photos. With that all requirements can be fulfilled. But by using a computer a new requirement got added: the screen should be locked.

This requirement created a challenge. As the maintainer of Plasma’s lock screen infrastructure I know what it is good at and that is blocking input and preventing other applications to be visible. Thus we cannot just use e.g. Digikam with gphoto2 integration to take the photo – the lock screen would prevent the Digikam window to be visible. Also there is no way to have a slide show in the lock screen.

Which means all requirements must be fulfilled through the lock screen infrastructure. A result of that is that I spent some time on adding wallpaper plugin support to the lock screen. This allowed to reuse Plasma’s wallpaper plugins and thus also the slide show plugin. One problem solved and all Plasma users can benefit from it.

But how to trigger gphoto2? The idea I came up with is using KWin/Wayland. KWin has full control over the input stack (even before the lock screen intercepts) and also knows which input devices it is interacting with. As a remote control I decided to use a Logitech Presenter and accept any clicked button on that device as the trigger. The code looks like the following:

class PhotoBoxFilter : public InputEventFilter { public: PhotoBoxFilter() : InputEventFilter() , m_tetheredProcess(new Process) { m_tetheredProcess->setProcessChannelMode(QProcess::ForwardedErrorChannel); m_tetheredProcess->setWorkingDirectory(QStringLiteral("/path/to/storage")); m_tetheredProcess->setArguments(QStringList{ QStringLiteral("--set-config"), QStringLiteral("output=TFT"), QStringLiteral("--capture-image-and-download"), QStringLiteral("--filename"), QStringLiteral("%Y%m%d-%H%M%S.jpg"), QStringLiteral("-I"), QStringLiteral("-1"), QStringLiteral("--reset-interval") }); m_tetheredProcess->setProgram(QStringLiteral("gphoto2")); } virtual ~PhotoBoxFilter() { m_tetheredProcess->terminate(); m_tetheredProcess->waitForFinished(); } bool keyEvent(QKeyEvent *event) override { if (!waylandServer()->isScreenLocked()) { return false; } auto key = static_cast(event); if (key->device() && key->device()->vendor() == 1133u && key->device()->product() == 50453u) { if (event->type() == QEvent::KeyRelease) { if (m_tetheredProcess->state() != QProcess::Running) { m_tetheredProcess->start(); m_tetheredProcess->waitForStarted(5000); } else { ::kill(m_tetheredProcess->pid(), SIGUSR1); } } return true; } return false; } private: QScopedPointer m_tetheredProcess; };

And in addition the method InputRedirection::setupInputFilters needed an adjustment to install this new InputFilter just before installing the LockScreenFilter.

The final setup:

  • Camera on tripod
  • Connected to an external screen showing the live capture
  • Connected to a notebook through USB
  • Notebook connected to an external TV
  • Notebook locked and lock screen configured to show slide show of photos
  • Logitech Presenter used as remote control

The last detail which needed adjustments was on the lock screen theme. The text input is destroying the experience of the slide show. Thus a small hack to the QML code was needed to hide it and reveal again after pointer motion.

What I want to show with this blog post is one of the advantage of open source software: you can adjust the software to your needs and turn it into something completely different which fits your needs.

Categories: FLOSS Project Planets

German Arias: Second release of eiffel-iup

GNU Planet! - Sun, 2016-08-21 01:54

I’m glad to announce the second release of eiffel-iup. A wrapper that allow create graphical applications with Liberty Eiffel using the IUP toolkit. This second version add flat buttons and fix some errors. The main changes are in the names of some features, which now have names in the eiffel style. This is enough mature to create graphical interfaces. The package contains examples that show how use eiiffel-iup. So let me know if you have problems and Happy hacking!


Categories: FLOSS Project Planets

diffutils @ Savannah: diffutils-3.5 released [stable]

GNU Planet! - Sun, 2016-08-21 01:41
This is to announce diffutils-3.5, a stable release. There have been 12 commits by 3 people in the 13 days since 3.4. See the NEWS below for a brief summary. Thanks to everyone who has contributed! The following people contributed changes to this release: Bastian Beischer (1) Hanno Böck (1) Jim Meyering (10) Jim [on behalf of the diffutils maintainers] ================================================================== Here is the GNU diffutils home page: http://gnu.org/s/diffutils/ For a summary of changes and contributors, see: http://git.sv.gnu.org/gitweb/?p=diffutils.git;a=shortlog;h=v3.5 or run this command from a git-cloned diffutils directory: git shortlog v3.4..v3.5 To summarize the 5 gnulib-related changes, run these commands from a git-cloned diffutils directory: git checkout v3.5 git submodule summary v3.4 Here are the compressed sources and a GPG detached signature[*]: http://ftp.gnu.org/gnu/diffutils/diffutils-3.5.tar.xz http://ftp.gnu.org/gnu/diffutils/diffutils-3.5.tar.xz.sig Use a mirror for higher download bandwidth: http://ftpmirror.gnu.org/diffutils/diffutils-3.5.tar.xz http://ftpmirror.gnu.org/diffutils/diffutils-3.5.tar.xz.sig [*] Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this: gpg --verify diffutils-3.5.tar.xz.sig If that command fails because you don't have the required public key, then run this command to import it: gpg --keyserver keys.gnupg.net --recv-keys 7FD9FCCB000BEEEE and rerun the 'gpg --verify' command. This release was bootstrapped with the following tools: Autoconf 2.69.147-5ad35 Automake 1.99a Gnulib v0.1-825-g1eb82ad ================================================================== NEWS * Noteworthy changes in release 3.5 (2016-08-20) [stable] ** Bug fixes diff3 no longer malfunctions due to use-after-free [bug introduced in 3.4] diff --color no longer colorizes when TERM=dumb
Categories: FLOSS Project Planets

Full Stack Python: Python for Entrepreneurs

Planet Python - Sun, 2016-08-21 00:00

Python for Entrepreneurs is a new video course by the creators of Talk Python to Me and Full Stack Python.

We are creating this course and running a Kickstarter for it based on feedback that it's still too damn difficult to turn basic Python programming knowledge into a business to generate income as a side or full time project. Both Michael and I have been able to make that happen for ourselves and we want to share every difficult lesson we've learned through this course.

The Python for Entrepreneurs videos and content will dive into building and deploying a real-world web application, marketing it to prospective customers, handling search engine optimization, making money through credit card payments, getting help from part-time contractors for niche tasks and scaling up to meet traffic demands.

If this course hits the mark for what you want to do with Python, check out the Kickstarter - we've set up steep discounts for early backers.

If you have any questions, please reach out to Michael Kennedy or me, Matt Makai.

Categories: FLOSS Project Planets

Danny Englander: Drupal 8 Development: How to Import an Existing Site Configuration Into a New Site

Planet Drupal - Sat, 2016-08-20 20:31

Disclaimer: The steps in this tutorial are not recommend for a live site and are experimental!

Recently, I ran into an issue in my local Drupal 8 development environment where uninstalling a module failed. I was getting errors related to the module in question in both the UI and in drush. My bad for not having a recent DB backup for my local, the one I had was too old to use. (I've since fixed that with an automated backup script!) Double my bad for having installed a not-ready-for-primetime module.

That being said, my local site had a ton of configuration and custom entities and I really wanted a clean database. I first attempted to use Features to transport all my config to a new install but kept getting an "Unmet Dependancies" error and was not able to get to the bottom of it. (There are a number of Drupal 8 Features issues open related to this.) I didn't understand this error as I was simply using the same root folder with a new database.

Features aside, I knew that by nature, an existing site configuration export can't be imported into another site, this makes sense at face value. But I got to thinking, "well, I just want to take all this nice config and pop it into a brand new site" -- less anything relating to the aforementioned bad module.

Get the site UUID

I did more searching and sure enough there is roundabout way to do what I was intending. It involves a few drush commands, drush cget and drush cset. The first command is to get the existing UUID from your present site.

drush cget system.site uuid

This will print out the Site UUID as:

'system.site:uuid': bfb11978-d1a3-4eda-91fb-45decf134e25

Once you get this, be sure to run drush cex one last time which will export the current configuration.

Install and reset

Once I had the current UUID, I put my settings files back to default, created a new database and installed Drupal 8 from scratch. Once the new site installed, I updated my settings.local.php to point to my existing configuration files:

/sites/default/files/config_......./sync/

If your remote is on Pantheon, the local config path would be like this:

/sites/default/files/config/

Once this was set, all I had to do was reset the new site UUID to my old one that I derived from the drush cget command by running this:

drush cset system.site uuid bfb11978-d1a3-4eda-91fb-45decf134e25

This resets the new site's UIID to the old site's UUID and that will allow you to run your config import.

Import the existing config

Now it was time to run drush cim which imports your site configuration. Running the command the first time gave me a rather nasty looking error.

The import failed due for the following reasons: [error] Entities exist of type Shortcut link and Default. These entities need to be deleted before importing.

This might seem like a scary error but it just has to do with admin Shortcut links and there is a core issue open for this. At any rate this was a new site so I didn't really care about deleting these. I did more searching and discovered an obscure drush command to "fix" this:

drush ev '\Drupal::entityManager()->getStorage("shortcut_set")->load("default")->delete();'

Once I did that, my configuration imported like a charm, all 300 config files and several dozens of entities. I didn't see any errors after that, and everything was perfect now.

Summary

I am not sure there would be many use cases for doing what is outlined here but it did solve a specific issue, your milage may vary. I would be suspect of using this method for an existing site that has lots of things already built out. Keep in mind, I imported my existing config into a brand new site install.

Tags 
  • Drupal
  • DevOps
  • Drupal 8
  • Drush
  • Drupal Planet
Resources 
Categories: FLOSS Project Planets

Daniel Stender: Collected notes from Python packaging

Planet Debian - Sat, 2016-08-20 19:17

Here are some collected notes on some particular problems from packaging Python stuff for Debian, and more are coming up like this in the future. Some of the issues discussed here might be rather simple and even benign for the experienced packager, but maybe this is be helpful for people coming across the same issues for the first time, wondering what's going wrong. But some of the things discussed aren't easy. Here are the notes for this posting, there is no particular order.

UnicodeDecoreError on open() in Python 3 running in non-UTF-8 environments

I've came across this problem again recently packaging httpbin 0.5.0. The build breaks the following way e.g. trying to build with sbuild in a chroot, that's the first run of setup.py with the default Python 3 interpreter:

I: pybuild base:184: python3.5 setup.py clean Traceback (most recent call last): File "setup.py", line 5, in <module> os.path.join(os.path.dirname(__file__), 'README.rst')).read() File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2386: ordinal not in range(128) E: pybuild pybuild:274: clean: plugin distutils failed with: exit code=1: python3.5 setup.py clean

One comes across UnicodeDecodeError-s quite oftenly on different occasions, mostly related to Python 2 packaging (like here). Here it's the case that in setup.py the long_description is tried to be read from the opened UTF-8 encoded file README.rst:

long_description = open( os.path.join(os.path.dirname(__file__), 'README.rst')).read()

This is a problem for Python 3.5 (and other Python 3 versions) when setup.py is executed by an interpreter run in a non-UTF-8 environment1:

$ LANG=C.UTF-8 python3.5 setup.py clean running clean $ LANG=C python3.5 setup.py clean Traceback (most recent call last): File "setup.py", line 5, in <module> os.path.join(os.path.dirname(__file__), 'README.rst')).read() File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2386: ordinal not in range(128) $ LANG=C python2.7 setup.py clean running clean

The reason for this error is, the default encoding for file object returned by open() e.g. in Python 3.5 is platform dependent, so that read() fails on that when there's a mismatch:

>>> readme = open('README.rst') >>> readme <_io.TextIOWrapper name='README.rst' mode='r' encoding='ANSI_X3.4-1968'>

Non-UTF-8 build environments because $LANG isn't particularly set at all or set to C are common or even default in Debian packaging, like in the continuous integration resp. test building for reproducible builds the primary environment features that (see here). That's also true for the base system of the sbuild environment:

$ schroot -d / -c unstable-amd64-sbuild -u root (unstable-amd64-sbuild)root@varuna:/# locale LANG= LANGUAGE= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX" LC_ALL=

A problem like this is solved mostly elegant by installing some workaround in debian/rules. A quick and easy fix is to add export LC_ALL=C.UTF-8 here, which supersedes the locale settings of the build environment. $LC_ALL is commonly used to change the existing locale settings, it overrides all other locale variables with the same value (see here). C.UTF-8 is an UTF-8 locale which is available by default in a base system, it could be used without installing the locales package (or worse, the huge locales-all package):

(unstable-amd64-sbuild)root@varuna:/# locale -a C C.UTF-8 POSIX

This problem ideally should be taken care of upstream. In Python 3, the default open() is io.open(), for which the specific encoding could be given, so that the UnicodeDecodeError vanishes. Python 2 uses os.open() for open(), which doesn't support that, but has io.open(), too. A fix which works for both Python branches goes like this:

import io long_description = io.open( os.path.join(os.path.dirname(__file__), 'README.rst'), encoding='utf-8').read() non-deterministic order of requirements in egg-info/requires.txt

This problem appeared in prospector/0.11.7-5 in the reproducible builds test builds, that was the first package of Prospector running on Python 32. It was revealed that there were differences in the sorting order of the [with_everything] dependencies resp. requirements in prospector-0.11.7.egg-info/requires.txt if the package was build on varying systems:

$ debbindiff b1/prospector_0.11.7-6_amd64.changes b2/prospector_0.11.7-6_amd64.changes {...} ├── prospector_0.11.7-6_all.deb │ ├── file list │ │ @@ -1,3 +1,3 @@ │ │ -rw-r--r-- 0 0 0 4 2016-04-01 20:01:56.000000 debian-binary │ │ --rw-r--r-- 0 0 0 4343 2016-04-01 20:01:56.000000 control.tar.gz │ │ +-rw-r--r-- 0 0 0 4344 2016-04-01 20:01:56.000000 control.tar.gz │ │ -rw-r--r-- 0 0 0 74512 2016-04-01 20:01:56.000000 data.tar.xz │ ├── control.tar.gz │ │ ├── control.tar │ │ │ ├── ./md5sums │ │ │ │ ├── md5sums │ │ │ │ │┄ Files in package differ │ ├── data.tar.xz │ │ ├── data.tar │ │ │ ├── ./usr/share/prospector/prospector-0.11.7.egg-info/requires.txt │ │ │ │ @@ -1,12 +1,12 @@ │ │ │ │ │ │ │ │ [with_everything] │ │ │ │ +pyroma>=1.6,<2.0 │ │ │ │ frosted>=1.4.1 │ │ │ │ vulture>=0.6 │ │ │ │ -pyroma>=1.6,<2.0

Reproducible builds folks recognized this to be a toolchain problem and set up the issue randonmness_in_python_setuptools_requires.txt to cover this. Plus, a wishlist bug against python-setuptools was filed to fix this. The patch which was provided by Chris Lamb adds sorting of dependencies in requires.txt for Setuptools by adding sorted() (stdlib) to _write_requirements() in command/egg_info.py:

--- a/setuptools/command/egg_info.py +++ b/setuptools/command/egg_info.py @@ -406,7 +406,7 @@ def warn_depends_obsolete(cmd, basename, filename): def _write_requirements(stream, reqs): lines = yield_lines(reqs or ()) append_cr = lambda line: line + '\n' - lines = map(append_cr, lines) + lines = map(append_cr, sorted(lines)) stream.writelines(lines)

O.k. In the toolchain, no instance sorts these requirements properly if differences appear, but what is the reason for these differences in the Prospector packages, though? The problem is somewhat subtle. In setup.py, [with_everything] is a dictionary entry of _OPTIONAL (which is used for extras_require) that is created by a list comprehension out of the other values in that dictionary. The code goes like this:

_OPTIONAL = { 'with_frosted': ('frosted>=1.4.1',), 'with_vulture': ('vulture>=0.6',), 'with_pyroma': ('pyroma>=1.6,<2.0',), 'with_pep257': (), # note: this is no longer optional, so this option will be removed in a future release } _OPTIONAL['with_everything'] = [req for req_list in _OPTIONAL.values() for req in req_list]

The result, the new _OPTIONAL dictionary including the key with_everything (which w/o further sorting is the source of the list of requirements requires.txt) e.g. looks like this (code snipped run through IPython):

In [3]: _OPTIONAL Out[3]: {'with_everything': ['vulture>=0.6', 'pyroma>=1.6,<2.0', 'frosted>=1.4.1'], 'with_vulture': ('vulture>=0.6',), 'with_pyroma': ('pyroma>=1.6,<2.0',), 'with_frosted': ('frosted>=1.4.1',), 'with_pep257': ()}

That list comprehension iterates over the other dictionary entries to gather the value of with_everything, and – Python programmers know that of course – dictionaries are not indexed and therefore the order of the key-value pairs isn't fixed, but is determined by certain conditions from outside the interpreter. That's the source for the non-determinism of this Debian package revision of Prospector.

This problem has been fixed by a patch, which just presorts the list of requirements before it gets added to _OPTIONAL:

@@ -76,8 +76,8 @@ 'with_pyroma': ('pyroma>=1.6,<2.0',), 'with_pep257': (), # note: this is no longer optional, so this option will be removed in a future release } -_OPTIONAL['with_everything'] = [req for req_list in _OPTIONAL.values() for req in req_list] - +with_everything = [req for req_list in _OPTIONAL.values() for req in req_list] +_OPTIONAL['with_everything'] = sorted(with_everything)

In comparison to the list method sort(), the function sorted() does not change iterables in-place, but returns a new object, both could be used. As a side note, egg-info/requires.txt isn't even needed, but that's another issue.

  1. As an alternative to prefixing LC_ALL, env -i could be used to get an empty environment. 

  2. 0.11.7-4 already but this package revision was in experimental due to the switch to Python 3 and therefore not tested by reproducible builds continuous integration. 

Categories: FLOSS Project Planets

Francois Marier: Remplacer un disque RAID défectueux

Planet Debian - Sat, 2016-08-20 18:00

Traduction de l'article original anglais à https://feeding.cloud.geek.nz/posts/replacing-a-failed-raid-drive/.

Voici la procédure que j'ai suivi pour remplacer un disque RAID défectueux sur une machine Debian.

Remplacer le disque

Après avoir remarqué que /dev/sdb a été expulsé de mon RAID, j'ai utilisé smartmontools pour identifier le numéro de série du disque à retirer :

smartctl -a /dev/sdb

Cette information en main, j'ai fermé l'ordinateur, retiré le disque défectueux et mis un nouveau disque vide à la place.

Initialiser le nouveau disque

Après avoir démarré avec le nouveau disque vide, j'ai copié la table de partitions avec parted.

Premièrement, j'ai examiné la table de partitions sur le disque dur non-défectueux :

$ parted /dev/sda unit s print

et créé une nouvelle table de partitions sur le disque de remplacement :

$ parted /dev/sdb unit s mktable gpt

Ensuite j'ai utilisé la commande mkpart pour mes 4 partitions et je leur ai toutes donné la même taille que les partitions équivalentes sur /dev/sda.

Finalement, j'ai utilisé les commandes toggle 1 bios_grub (partition d'amorce) et toggle X raid (où X est le numéro de la partition) pour toutes les partitions RAID, avant de vérifier avec la commande print que les deux tables de partitions sont maintenant identiques.

Resynchroniser/recréer les RAID

Pour synchroniser les données du bon disque (/dev/sda) vers celui de remplacement (/dev/sdb), j'ai exécuté les commandes suivantes sur mes partitions RAID1 :

mdadm /dev/md0 -a /dev/sdb2 mdadm /dev/md2 -a /dev/sdb4

et j'ai gardé un oeil sur le statut de la synchronisation avec :

watch -n 2 cat /proc/mdstat

Pour accélérer le processus, j'ai utilisé le truc suivant :

blockdev --setra 65536 "/dev/md0" blockdev --setra 65536 "/dev/md2" echo 300000 > /proc/sys/dev/raid/speed_limit_min echo 1000000 > /proc/sys/dev/raid/speed_limit_max

Ensuite, j'ai recréé ma partition swap RAID0 comme suit :

mdadm /dev/md1 --create --level=0 --raid-devices=2 /dev/sda3 /dev/sdb3 mkswap /dev/md1

Par que la partition swap est toute neuve (il n'est pas possible de restorer une partition RAID0, il faut la re-créer complètement), j'ai dû faire deux choses:

  • remplacer le UUID pour swap dans /etc/fstab, avec le UUID donné par la commande mkswap (ou bien en utilisant la command blkid et en prenant le UUID pour /dev/md1)
  • remplacer le UUID de /dev/md1 dans /etc/mdadm/mdadm.conf avec celui retourné pour /dev/md1 par la commande mdadm --detail --scan
S'assurer que l'on peut démarrer avec le disque de remplacement

Pour être certain de bien pouvoir démarrer la machine avec n'importe quel des deux disques, j'ai réinstallé le boot loader grub sur le nouveau disque :

grub-install /dev/sdb

avant de redémarrer avec les deux disques connectés. Ceci confirme que ma configuration fonctionne bien.

Ensuite, j'ai démarré sans le disque /dev/sda pour m'assurer que tout fonctionnerait bien si ce disque décidait de mourir et de me laisser seulement avec le nouveau (/dev/sdb).

Ce test brise évidemment la synchronisation entre les deux disques, donc j'ai dû redémarrer avec les deux disques connectés et puis ré-ajouter /dev/sda à tous les RAID1 :

mdadm /dev/md0 -a /dev/sda2 mdadm /dev/md2 -a /dev/sda4

Une fois le tout fini, j'ai redémarrer à nouveau avec les deux disques pour confirmer que tout fonctionne bien :

cat /proc/mdstat

et j'ai ensuite exécuter un test SMART complet sur le nouveau disque :

smartctl -t long /dev/sdb
Categories: FLOSS Project Planets

Francois Marier: Remplacer un disque RAID défectueux

Planet Debian - Sat, 2016-08-20 18:00

Traduction de l'article original anglais à https://feeding.cloud.geek.nz/posts/replacing-a-failed-raid-drive/.

Voici la procédure que j'ai suivi pour remplacer un disque RAID défectueux sur une machine Debian.

Remplacer le disque

Après avoir remarqué que /dev/sdb a été expulsé de mon RAID, j'ai utilisé smartmontools pour identifier le numéro de série du disque à retirer :

smartctl -a /dev/sdb

Cette information en main, j'ai fermé l'ordinateur, retiré le disque défectueux et mis un nouveau disque vide à la place.

Initialiser le nouveau disque

Après avoir démarré avec le nouveau disque vide, j'ai copié la table de partitions avec parted.

Premièrement, j'ai examiné la table de partitions sur le disque dur non-défectueux :

$ parted /dev/sda unit s print

et créé une nouvelle table de partitions sur le disque de remplacement :

$ parted /dev/sdb unit s mktable gpt

Ensuite j'ai utilisé la commande mkpart pour mes 4 partitions et je leur ai toutes donné la même taille que les partitions équivalentes sur /dev/sda.

Finalement, j'ai utilisé les commandes toggle 1 bios_grub (partition d'amorce) et toggle X raid (où X est le numéro de la partition) pour toutes les partitions RAID, avant de vérifier avec la commande print que les deux tables de partitions sont maintenant identiques.

Resynchroniser/recréer les RAID

Pour synchroniser les données du bon disque (/dev/sda) vers celui de remplacement (/dev/sdb), j'ai exécuté les commandes suivantes sur mes partitions RAID1 :

mdadm /dev/md0 -a /dev/sdb2 mdadm /dev/md2 -a /dev/sdb4

et j'ai gardé un oeil sur le statut de la synchronisation avec :

watch -n 2 cat /proc/mdstat

Pour accélérer le processus, j'ai utilisé le truc suivant :

blockdev --setra 65536 "/dev/md0" blockdev --setra 65536 "/dev/md2" echo 300000 > /proc/sys/dev/raid/speed_limit_min echo 1000000 > /proc/sys/dev/raid/speed_limit_max

Ensuite, j'ai recréé ma partition swap RAID0 comme suit :

mdadm /dev/md1 --create --level=0 --raid-devices=2 /dev/sda3 /dev/sdb3 mkswap /dev/md1

Par que la partition swap est toute neuve (il n'est pas possible de restorer une partition RAID0, il faut la re-créer complètement), j'ai dû faire deux choses:

  • remplacer le UUID pour swap dans /etc/fstab, avec le UUID donné par la commande mkswap (ou bien en utilisant la command blkid et en prenant le UUID pour /dev/md1)
  • remplacer le UUID de /dev/md1 dans /etc/mdadm/mdadm.conf avec celui retourné pour /dev/md1 par la commande mdadm --detail --scan
S'assurer que l'on peut démarrer avec le disque de remplacement

Pour être certain de bien pouvoir démarrer la machine avec n'importe quel des deux disques, j'ai réinstallé le boot loader grub sur le nouveau disque :

grub-install /dev/sdb

avant de redémarrer avec les deux disques connectés. Ceci confirme que ma configuration fonctionne bien.

Ensuite, j'ai démarré sans le disque /dev/sda pour m'assurer que tout fonctionnerait bien si ce disque décidait de mourir et de me laisser seulement avec le nouveau (/dev/sdb).

Ce test brise évidemment la synchronisation entre les deux disques, donc j'ai dû redémarrer avec les deux disques connectés et puis ré-ajouter /dev/sda à tous les RAID1 :

mdadm /dev/md0 -a /dev/sda2 mdadm /dev/md2 -a /dev/sda4

Une fois le tout fini, j'ai redémarrer à nouveau avec les deux disques pour confirmer que tout fonctionne bien :

cat /proc/mdstat

et j'ai ensuite exécuter un test SMART complet sur le nouveau disque :

smartctl -t long /dev/sdb
Categories: FLOSS Project Planets

Roy Scholten: Getting something in the box

Planet Drupal - Sat, 2016-08-20 17:53

First impressions matter. The first glance has a lot if impact on further expectations. Drupal core doesn’t do well there. As webchick points out, after installation the opening line is “you have no content”.

Yeah, thanks.

This empty canvas makes Drupal appear very limited in functionality. Which is the exact opposite of how Drupal is advertised (flexible, extensible, there’s a module for that!)

This is not news. The issue for adding sample content is over 10 years old. The image I posted earlier is from a core conversation 6 years ago. Eaton and moi presented on Snowman in Prague 3 years ago.

But now we do have Drupal 8, with essential features available right out of the box. We have a new release schedule that encourages shipping new features and improvements every 6 months. And we’re settling on a better process for figuring out the part from initial idea to fleshed out plan first, before implementing that plan.

So lets work together and come up with a plan for sample content in core. Which means answering product focussed questions like:

  • Audience: who do we make this for?
  • Goals: what do these people want to achieve?
  • Features: which tools (features) to provide to make that possible?
  • Priorities: which of those tools are essential, which are nice to haves?

But purpose first: audience and goals.

We’re always balancing product specifics with framework generics in what core provides. Pretty sure we can do something more opiniated than our current default “Article” and “Page” content types without painting ourselves in a corner.

We’ll discuss this topic during upcoming UX meetings and in the UX channel in drupal.slack.com (get your automatic invite here).

Tags: drupalonboardingdrupalplanetSub title: Tabula rasa is not an effective onboarding strategy
Categories: FLOSS Project Planets

Podcast.__init__: Episode 71 - GenSim with Radim Řehůřek

Planet Python - Sat, 2016-08-20 15:09
Summary

Being able to understand the context of a piece of text is generally thought to be the domain of human intelligence. However, topic modeling and semantic analysis can be used to allow a computer to determine whether different messages and articles are about the same thing. This week we spoke with Radim Řehůřek about his work on GenSim, which is a Python library for performing unsupervised analysis of unstructured text and applying machine learning models to the problem of natural language understanding.

Brief Introduction
  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com
  • Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project
  • We are also sponsored by Sentry this week. Stop hoping your users will report bugs. Sentry’s real-time tracking gives you insight into production deployments and information to reproduce and fix crashes. Check them out at getsentry.com and use the code podcastinit at signup to get a $50 credit on your account.
  • Visit our site to subscribe to our show, sign up for our newsletter, read the show notes, and get in touch.
  • To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
  • Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.
  • Your hosts as usual are Tobias Macey and Chris Patti
  • Today we’re interviewing Radim Řehůřek about Gensim, a library for topic modeling and semantic analysis of natural language.

Use the promo code podcastinit20 to get a $20 credit when you sign up!

Stop hoping your users will report bugs. Sentry’s real-time tracking gives you insight into production deployments and information to reproduce and fix crashes. Use the code podcastinit at signup to get a $50 credit!

Interview with Radim Řehůřek
  • Introductions
  • How did you get introduced to Python? - Chris
  • Can you start by giving us an explanation of topic modeling and semantic analysis? - Tobias
  • What is Gensim and what inspired you to create it? - Tobias
  • What facilities does Gensim provide to simplify the work of this kind of language analysis? - Tobias
  • Can you describe the features that set it apart from other projects such as the NLTK or Spacy? - Tobias
  • What are some of the practical applications that Gensim can be used for? - Tobias
  • One of the features that stuck out to me is the fact that Gensim can process corpora on disk that would be too large to fit into memory. Can you explain some of the algorithmic work that was necessary to allow for this streaming process to be possible? - Tobias
    • Given that it can handle streams of data, could it also be used in the context of something like Spark? - Tobias
  • Gensim also supports unsupervised model building. What kinds of limitations does this have and when would you need a human in the loop? - Tobias
    • Once a model has been trained, how does it get saved and reloaded for subsequent use? - Tobias
  • What are some of the more unorthodox or interesting uses people have put Gensim to that you’ve heard about? - Chris
  • In addition to your work on Gensim, and partly due to its popularity, you have started a consultancy for customers who are interested in improving their data analysis capabilities. How does that feed back into Gensim? - Tobias
  • Are there any improvements in Gensim or other libraries that you have made available as a result of issues that have come up during client engagements? - Tobias
  • Is it difficult to find contributors to Gensim because of its advanced nature? - Tobias
  • Are there any resources you’d like to recommend our listeners explore to get a more in depth understanding of topic modeling and related techniques? - Chris
Keep In Touch Picks Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Summary Being able to understand the context of a piece of text is generally thought to be the domain of human intelligence. However, topic modeling and semantic analysis can be used to allow a computer to determine whether different messages and articles are about the same thing. This week we spoke with Radim Řehůřek about his work on GenSim, which is a Python library for performing unsupervised analysis of unstructured text and applying machine learning models to the problem of natural language understanding.Brief IntroductionHello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.comLinode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next projectWe are also sponsored by Sentry this week. Stop hoping your users will report bugs. Sentry's real-time tracking gives you insight into production deployments and information to reproduce and fix crashes. Check them out at getsentry.com and use the code podcastinit at signup to get a $50 credit on your account.Visit our site to subscribe to our show, sign up for our newsletter, read the show notes, and get in touch.To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workersJoin our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas.Your hosts as usual are Tobias Macey and Chris PattiToday we're interviewing Radim Řehůřek about Gensim, a library for topic modeling and semantic analysis of natural language. Use the promo code podcastinit20 to get a $20 credit when you sign up! Stop hoping your users will report bugs. Sentry's real-time tracking gives you insight into production deployments and information to reproduce and fix crashes. Use the code podcastinit at signup to get a $50 credit!Interview with Radim ŘehůřekIntroductionsHow did you get introduced to Python? - ChrisCan you start by giving us an explanation of topic modeling and semantic analysis? - TobiasWhat is Gensim and what inspired you to create it? - TobiasWhat facilities does Gensim provide to simplify the work of this kind of language analysis? - TobiasCan you describe the features that set it apart from other projects such as the NLTK or Spacy? - TobiasWhat are some of the practical applications that Gensim can be used for? - TobiasOne of the features that stuck out to me is the fact that Gensim can process corpora on disk that would be too large to fit into memory. Can you explain some of the algorithmic work that was necessary to allow for this streaming process to be possible? - TobiasGiven that it can handle streams of data, could it also be used in the context of something like Spark? - TobiasGensim also supports unsupervised model building. What kinds of limitations does this have and when would you need a human in the loop? - TobiasOnce a model has been trained, how does it get saved and reloaded for subsequent use? - TobiasWhat are some of the more unorthodox or interesting uses people have put Gensim to that you've heard about? - ChrisIn addition to your work on Gensim, and partly due to its popularity, you have started a consultancy for customers who are interested in improving their data analysis capabilities. How does that feed back into Gensim? - TobiasAre there any improvements in Gensim or other libraries that you have made available as a result of issues that have come up during client engagements? - TobiasIs it difficult to find contributors to Gensim because of its advanced nature? - TobiasAre there any resources you'd like to recommend our listeners explore to get a more in depth understanding of topic m
Categories: FLOSS Project Planets

Falit Jain (falitjain)

Planet KDE - Sat, 2016-08-20 14:42

Categories: FLOSS Project Planets

The Editor Improved!!

Planet KDE - Sat, 2016-08-20 14:42

It's been 4 months , I have been associated with WikiToLearn . 
The experience : Awesome , awesome and awesome.So much learning in so little time , connecting with devs across the world , being associated to a community ,bootstrapping my open-source contribution, yes , that's what I wanted from GSoC :D
Mentors and the ProjectIrene Cortinovis and Gianluca Rigoletti . These two have been answering to every query , be it      trivial or not from the very starting to till now. Every time , they responded ,and unblocked me fromthe tasks. They assigned me tasks in a very planned manner. Lucky , to have mentor such as them.
What I enjoy the most , is the tech group , where all discussions takes place. Every time someone asks anything , it's fun to discuss on a common group , it makes everyone learn something . Bots too add the fun element to the discussions.
Project : Adding the Latex Completions for the Visual Editor
The earlier editor didn't had any autocompletions whether in adding code blocks or inserting formula:

That Improved by Adding Autocompletions and Latex Snippets:
Snippets and AutoCompletions have now become a part of the editor. The user need not to write complex formulas completely. Corresponding snippet's caption can be used to render the formula with ease. Other than , the text already written can be autocompleted by a dynamically generated dropdown.
Validation of Security from XSS Attacks:When adding <a href=""> tags to the source directly:


When trying to exploite the style attribute of the div:

It was detected as an insecure input :) :)


Complete Documentation:https://www.mediawiki.org/wiki/Extension:LatexCompleterhttps://community.kde.org/GSoC/2016/StatusReports/FalitJain
Work Remaining:Some of the XSS tests on IE are still remaining. Things I learned :
  • Communicate properly and clearly .
  • Implement keeping the future in mind.
  • VIM and GIT concepts refined
  • Insight to open source
  • People from Italy  :) ;)
Things I loved:
  • The TECH CHANNEL .
  • Building end user experience
  • Working through VIM
Thanks again to Irene and Gianluca for helping me till now . WikiToLearn is a fun family , which I am lucky to be a part of , and hopefully we will keep growing .
    KNOWLEDGE ONLY GROWS IF SHARED.
    Categories: FLOSS Project Planets

    Jose M. Calhariz: Availabilty of at at the Major Linux Distributions

    Planet Debian - Sat, 2016-08-20 14:11

    In this blog post I will cover what versions of software at is used by the leading Linux Distributions as reported by LWN.

    Also

    Currently some distributions are lagging on the use of the latest at software.

    Categories: FLOSS Project Planets

    Ian Ozsvald: Some notes on building a conda recipe

    Planet Python - Sat, 2016-08-20 11:32

    I’ve spent the day building a conda recipe, the process wasn’t super-smooth, hopefully these notes will help others and/or maybe you can leave me a comment to improve my flow. The goal was to learn how to use conda to distribute a package that ordinarily I’d put on PyPI.

    I’m using Linux 64bit (Mint 18 on an XPS 9550), conda 4.1 and conda build 1.21.14 (up to date as of today). My goal was to build a recipe (python_template_with_config_recipe) to install my python_template_with_config (a bit of boilerplate code I use sometimes when making a new project). That template has two modules, a test, a setup.py and it depends on numpy.

    The short story:

    1. git clone https://github.com/ianozsvald/python_template_with_config_recipe.git
    2. cd inside, run “conda build –debug .”
    3. # note the period means “current directory” and that’s two dashes debug
    4. a local bzip will be built and  you’ll see that the 1 test ran ok

    On my machine the built code ends up in “~/anaconda3/pkgs/python_template_with_config-0.1-py35_0/lib/python3.5/site-packages/python_template_with_config” and the building takes place in “~/anaconda3/conda-bld/linux-64”.

    In a new conda environment I can use “conda install –use-local python_template_with_config” and it’ll install the built recipe into the new environment.

    To get started with this I first made a fresh empty conda environment (note that anaconda isn’t my default Python, hence the long-form access to `conda`):

    1. $ ~/anaconda3/bin/conda create -n new_recipe_env python=3.5
    2. $ . ~/anaconda3/bin/activate new_recipe_env

    To check that my existing “setup.py” runs I use pip to install from git, we’ll need the setup.py in the conda recipe later so we want to confirm that it works:

    • $ pip install git+https://github.com/ianozsvald/python_template_with_config.git # runs setup.py
    • # $ pip uninstall python_template_with_config # use this if you need to uninstall whilst developing

    I can check that this has installed as a module using:

    In [1]: from python_template_with_config import another_module In [2]: another_module.a_math_function() # silly function just to check that numpy is installed Out[2]: -2.4492935982947064e-16

    Now I’ll make a second conda environment to develop the recipe:

    1. $ ~/anaconda3/bin/conda create -n new_recipe_env2 python=3.5 # vanilla environment, no numpy
    2. $ . ~/anaconda3/bin/activate new_recipe_env2
    3. git clone https://github.com/ianozsvald/python_template_with_config_recipe.git
    4. cd inside, run "conda build --debug ."

    The recipe (meta.yaml) will look at the git repo for python_template_with_config, pull down a copy, build using build.sh and the store a bzip2 archive locally. The build step also notes that I can upload this to Anaconda using `$ anaconda upload /home/ian/anaconda3/conda-bld/linux-64/python_template_with_config-0.1-py35_0.tar.bz2`.

    A few caveats occurred whilst creating the recipe:

    • You need a bld.bat, build.sh, meta.yaml, at first I created bld.sh and meta.yml (both typos) and there were no complaints…just frustration on my part – the first clue was seeing “source tree  in: /home/ian/anaconda3/conda-bld/work \n number of files: 0” in the build output
    • When running conda build it seems to not overwrite the version in ~/anaconda3/pkgs/ – I ended up deleting “python_template_with_config-0.1-py35_0/” and “python_template_with_config-0.1-py35_0.tar.bz2” by hand just to make sure on each build iteration – I must be missing something here, please enlighten me (see note from Marco below)
    • Having deleted the cached versions and fixed the typos I’d later see “number of files: 14”
    • Later I added “run_tests.py” rather than “run_test.py”, I knew it wasn’t running as I’d added a “1/0” line inside run_tests.py that obviously wasn’t running (it should raise a ZeroDivisionError even if the tests did run ok). Again this was a typo on my part
    • The above is tested on Linux, it ought to work on Windows but I’ve not tested it
    • This meta.yaml installs from github, there’s a commented out line in there showing how to access the local source files instead

    Marco Bonzanini has noted that “conda clean“, “conda clean -t” (tarballs) and “conda clean -p” (packages) can help with the caching issue mentioned above. He also notes “conda skeleton <pypi package url>” takes care of the boilerplate if you have a published version on PyPI, so that avoids the silly mistakes I made by hand. Cheers!

    I didn’t get as far as uploading this to Anaconda to make it ‘public’ (as I don’t think that’s so useful) but I believe that final step is easy enough.

    Useful docs:

    Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.
    Categories: FLOSS Project Planets

    BangPypers: Talks About Automation - 2016 August Meetup

    Planet Python - Sat, 2016-08-20 09:51

    August meetup happened at the Calm.io office. 60 people attended the event. For the first time, all talks were arranged in Automation theme. Thanks for the warm response from the audience and enthusiastic proposals from speakers.

    We had a hiccup with the projector and lost fifteen minutes, but meet up commenced on time.

    Praveen presented the first talk “Building and maintaining a self-developed test-suite”. You can find the slides of the session here.

    First talk of automation theme is in progress “Building and maintaining a self-developed test-suite” by Praveen. pic.twitter.com/xMIpBsxzbM

    — BangPypers (@__bangpypers__) August 20, 2016 The Second talk was by JoyDeep about “Windows task automation using pywinauto”. The crisp talk duration was fifteen minutes.

    Flash talk by Joydeep about “Windows automating with pywinauto” pic.twitter.com/49RvlgAj6V

    — BangPypers (@__bangpypers__) August 20, 2016 You can find Slides for the talk in [Google Drive](https://drive.google.com/file/d/0B_4FAm4Vg99kVWs1azd4SjVRZG8/view) and, the code snippet in [GitHub)(https://github.com/infinite-Joy/windows_automation). Calm offered us tea and snacks during the break. The participants took a break for twenty minutes. A lot of first timers and repeated audience indulged in offline discussions about the talks and their interests. After the refreshing break, Abraham presented “Setting development environment with Ansible”. It was a lively session with questions flying from all corners of the room, and the speaker answered all of the very well. We didn’t have enough time left for Q&A. Here is the link to [slides](https://docs.google.com/presentation/d/1UOIzsniQae64AX5zxYD8ttw7Je7jaulfdbipg7k-J8E/edit?usp=sharing).

    Setting up development environment using Ansible by Abhraham pic.twitter.com/eTNzJ6kt1S

    — BangPypers (@__bangpypers__) August 20, 2016

    The last talk of the day was from Srinath about “Building Scalable Server Automation with Python”. Srinath talked about the design drawbacks of their product over the years. How each version improved performance and how efficient is the current generation product. This talk was very well received and enabled discussion among the audience.

    You can find the slides in Google Drive.

    Last talk of the day, Building Scalable Server Automation with Python by balaji pic.twitter.com/8fpwtxuYV5

    — BangPypers (@__bangpypers__) August 20, 2016

    Please a take thirty seconds to provide us your feedback in this form. We highly appreciate your efforts to help us improve the meetup. We do meetups for people like you!

    In case you would like to host us in your premise, leave a note on our meetup page.

    Thanks to speakers for putting up a great show and, Calm.io for hosting the event. Hope you had a great time with us.

    See you in our next meetup.

    Categories: FLOSS Project Planets

    Towards an Efficient Linux Photographic Workflow

    Planet KDE - Sat, 2016-08-20 09:11

    digiKam is the cornerstone of my photographic workflow. This powerful and versatile photo management application has all tools and features necessary for transferring, organizing, processing, and managing photos, RAW files, and videos. But even though digiKam can handle practically any photographic task you throw at it, there is still room for optimizing and improving parts of the Linux-based photographic workflow. Continue reading

    Categories: FLOSS Project Planets

    It only takes a Minuet to change your life

    Planet KDE - Sat, 2016-08-20 07:52

    It has been so long since I posted, the reason is that I have been quite busy with my GSoC timeline. The GSoC period is about to end, but before I describe my GSoC journey and contributions, I would like to state that KDE will be launching the cool Minuet application for Android phones.

    What’s Minuet?

    Minuet music education Free Software, thoughtfully designed to push your musical skills to the next level. Enhance your ability to recognize intervals, chords, rhythm, and scales while enjoying all the power of Free Software.Yes it is open source

    The application is currently available on GNU/Linux and Android(pre Marshmallow). We will be soon releasing it on other platforms.

    Journey to GSoC

    My GsoC journey begun in mid-February, when I started contributing to Minuet repository of KDE. Fast forward two months, and I was thrilled to be selected for Google Summer of Code 2016. My main aim was to develop an Android application using Qt(C++) that had the same feature as that of the Desktop version.

    The project is currently being maintained by Sandro Andrade. He has been an amazing mentor and I am lucky to have him as my mentor

    I guess it is time to summarize the whole 3 months work. You can find my contributions here.

    I started the journey by looking for the cross platform MIDI libraries, believe me I had a tough time looking for these libraries until I came across Csound. Csound actually made my work easier by providing all the functionalities that I was looking for. Although on digging deep , we found out that Csound used Fluidsynth opcodes which again is a music synthesizer for Linux and currently used in Minuet desktop application. So may be in the near future we may use Fluidsynth in Minuet mobile application. The sound fonts currently being used is sf_GMbank.sf2.

    My next task was to implement Scales, Chords, Rhythms and Intervals exercises. Currently the exercise data is stored in JSON format. The advantage of using this method is that new exercises can be added with literally no changes in the code. So you can add your exercises too

    Once done with the above tasks, we(I and my mentor) had to design the UI of the application. Since smart phones are much smaller in size than desktops and laptops, our main challenge was to have an interactive user interface with  all the required elements in one page.

    After spending days working on the UI, we finally managed to come up with a working design.

    In the ending phase, we tried to fix all the bugs, make certain improvements and voila now finally we have a working Minuet Android application

    So this is how it looks so far.

    We have already sent a request to make the application available on Google Play Store. It should be available within a week or you can build it from here.

    For next year students?

    I have learned a lot from previous students/mentors hints. And as a GSoC participant duty, I should give next generation of students some kind of advice, so they never get surprised. That feedback wouldn’t differ a lot from what others would tell, write code a lot, keep strong bonds with your mentor.

    Keep asking questions, even if it is a silly one. Be patient on IRC also also keep contributing to the same organization even after GSoC period. The main aim of this program is to introduce students to open source projects.


    Categories: FLOSS Project Planets

    Robin Wilson: Regression in Python using R-style formula – it’s easy!

    Planet Python - Sat, 2016-08-20 06:54
    .highlight .hll {background-color:#ffffcc}.highlight {background:#e4e4e4;}.highlight .c {color:#408080;font-style:italic}.highlight .err {border:1px solid #FF0000}.highlight .k {color:#008000;font-weight:bold}.highlight .o {color:#666666}.highlight .cm {color:#408080;font-style:italic}.highlight .cp {color:#BC7A00}.highlight .c1 {color:#408080;font-style:italic}.highlight .cs {color:#408080;font-style:italic}.highlight .gd {color:#A00000}.highlight .ge {font-style:italic}.highlight .gr {color:#FF0000}.highlight .gh {color:#000080;font-weight:bold}.highlight .gi {color:#00A000}.highlight .go {color:#888888}.highlight .gp {color:#000080;font-weight:bold}.highlight .gs {font-weight:bold}.highlight .gu {color:#800080;font-weight:bold}.highlight .gt {color:#0044DD}.highlight .kc {color:#008000;font-weight:bold}.highlight .kd {color:#008000;font-weight:bold}.highlight .kn {color:#008000;font-weight:bold}.highlight .kp {color:#008000}.highlight .kr {color:#008000;font-weight:bold}.highlight .kt {color:#B00040}.highlight .m {color:#666666}.highlight .s {color:#BA2121}.highlight .na {color:#7D9029}.highlight .nb {color:#008000}.highlight .nc {color:#0000FF;font-weight:bold}.highlight .no {color:#880000}.highlight .nd {color:#AA22FF}.highlight .ni {color:#999999;font-weight:bold}.highlight .ne {color:#D2413A;font-weight:bold}.highlight .nf {color:#0000FF}.highlight .nl {color:#A0A000}.highlight .nn {color:#0000FF;font-weight:bold}.highlight .nt {color:#008000;font-weight:bold}.highlight .nv {color:#19177C}.highlight .ow {color:#AA22FF;font-weight:bold}.highlight .w {color:#bbbbbb}.highlight .mb {color:#666666}.highlight .mf {color:#666666}.highlight .mh {color:#666666}.highlight .mi {color:#666666}.highlight .mo {color:#666666}.highlight .sb {color:#BA2121}.highlight .sc {color:#BA2121}.highlight .sd {color:#BA2121;font-style:italic}.highlight .s2 {color:#BA2121}.highlight .se {color:#BB6622;font-weight:bold}.highlight .sh {color:#BA2121}.highlight .si {color:#BB6688;font-weight:bold}.highlight .sx {color:#008000}.highlight .sr {color:#BB6688}.highlight .s1 {color:#BA2121}.highlight .ss {color:#19177C}.highlight .bp {color:#008000}.highlight .vc {color:#19177C}.highlight .vg {color:#19177C}.highlight .vi {color:#19177C}.highlight .il {color:#666666}

    table, th, td {
    border: 0px
    }

    I remember experimenting with doing regressions in Python using R-style formulae a long time ago, and I remember it being a bit complicated. Luckily it’s become really easy now – and I’ll show you just how easy.

    Before running this you will need to install the pandas, statsmodels and patsy packages. If you’re using conda you should be able to do this by running the following from the terminal:

    conda install statsmodels patsy

    (and then say yes when it asks you to confirm it)

    import pandas as pd from statsmodels.formula.api import ols

    Before we can do any regression, we need some data – so lets read some data on cars:

    df = pd.read_csv("http://web.pdx.edu/~gerbing/data/cars.csv")

    You may have noticed from the code above that you can just give a URL to the read_csv function and it will download it and open it – handy!

    Anyway, here is the data:

    df.head() Model MPG Cylinders Engine Disp Horsepower Weight Accelerate Year Origin 0 amc ambassador dpl 15.0 8 390.0 190 3850 8.5 70 American 1 amc gremlin 21.0 6 199.0 90 2648 15.0 70 American 2 amc hornet 18.0 6 199.0 97 2774 15.5 70 American 3 amc rebel sst 16.0 8 304.0 150 3433 12.0 70 American 4 buick estate wagon (sw) 14.0 8 455.0 225 3086 10.0 70 American

    Before we do our regression it might be a good idea to look at simple correlations between columns. We can get the correlations between each pair of columns using the corr() method:

    df.corr() MPG Cylinders Engine Disp Horsepower Weight Accelerate Year MPG 1.000000 -0.777618 -0.805127 -0.778427 -0.832244 0.423329 0.580541 Cylinders -0.777618 1.000000 0.950823 0.842983 0.897527 -0.504683 -0.345647 Engine Disp -0.805127 0.950823 1.000000 0.897257 0.932994 -0.543800 -0.369855 Horsepower -0.778427 0.842983 0.897257 1.000000 0.864538 -0.689196 -0.416361 Weight -0.832244 0.897527 0.932994 0.864538 1.000000 -0.416839 -0.309120 Accelerate 0.423329 -0.504683 -0.543800 -0.689196 -0.416839 1.000000 0.290316 Year 0.580541 -0.345647 -0.369855 -0.416361 -0.309120 0.290316 1.000000

    Now we can do some regression using R-style formulae. In this case we’re trying to predict MPG based on the year that the car was released:

    model = ols("MPG ~ Year", data=df) results = model.fit()

    The ‘formula’ that we used above is the same as R uses: on the left is the dependent variable, on the right is the independent variable. The ols method is nice and easy, we just give it the formula, and then the DataFrame to use to get the data from (in this case, it’s called df). We then call fit() to actually do the regression.

    We can easily get a summary of the results here – including all sorts of crazy statistical measures!

    results.summary() OLS Regression Results Dep. Variable: MPG R-squared: 0.337 Model: OLS Adj. R-squared: 0.335 Method: Least Squares F-statistic: 198.3 Date: Sat, 20 Aug 2016 Prob (F-statistic): 1.08e-36 Time: 10:42:17 Log-Likelihood: -1280.6 No. Observations: 392 AIC: 2565. Df Residuals: 390 BIC: 2573. Df Model: 1 Covariance Type: nonrobust coef std err t P>|t| [95.0% Conf. Int.] Intercept -70.0117 6.645 -10.536 0.000 -83.076 -56.947 Year 1.2300 0.087 14.080 0.000 1.058 1.402 Omnibus: 21.407 Durbin-Watson: 1.121 Prob(Omnibus): 0.000 Jarque-Bera (JB): 15.843 Skew: 0.387 Prob(JB): 0.000363 Kurtosis: 2.391 Cond. No. 1.57e+03

    We can do a more complex model easily too. First lets list the columns of the data to remind us what variables we have:

    df.columns Index(['Model', 'MPG', 'Cylinders', 'Engine Disp', 'Horsepower', 'Weight', 'Accelerate', 'Year', 'Origin'], dtype='object')

    We can now add in more variables – doing multiple regression:

    model = ols("MPG ~ Year + Weight + Horsepower", data=df) results = model.fit() results.summary() OLS Regression Results Dep. Variable: MPG R-squared: 0.808 Model: OLS Adj. R-squared: 0.807 Method: Least Squares F-statistic: 545.4 Date: Sat, 20 Aug 2016 Prob (F-statistic): 9.37e-139 Time: 10:42:17 Log-Likelihood: -1037.4 No. Observations: 392 AIC: 2083. Df Residuals: 388 BIC: 2099. Df Model: 3 Covariance Type: nonrobust coef std err t P>|t| [95.0% Conf. Int.] Intercept -13.7194 4.182 -3.281 0.001 -21.941 -5.498 Year 0.7487 0.052 14.365 0.000 0.646 0.851 Weight -0.0064 0.000 -15.768 0.000 -0.007 -0.006 Horsepower -0.0050 0.009 -0.530 0.597 -0.024 0.014 Omnibus: 41.952 Durbin-Watson: 1.423 Prob(Omnibus): 0.000 Jarque-Bera (JB): 69.490 Skew: 0.671 Prob(JB): 8.14e-16 Kurtosis: 4.566 Cond. No. 7.48e+04

    We can see that bringing in some extra variables has increased the $R^2$ value from ~0.3 to ~0.8 – although we can see that the P value for the Horsepower is very high. If we remove Horsepower from the regression then it barely changes the results:

    model = ols("MPG ~ Year + Weight", data=df) results = model.fit() results.summary() OLS Regression Results Dep. Variable: MPG R-squared: 0.808 Model: OLS Adj. R-squared: 0.807 Method: Least Squares F-statistic: 819.5 Date: Sat, 20 Aug 2016 Prob (F-statistic): 3.33e-140 Time: 10:42:17 Log-Likelihood: -1037.6 No. Observations: 392 AIC: 2081. Df Residuals: 389 BIC: 2093. Df Model: 2 Covariance Type: nonrobust coef std err t P>|t| [95.0% Conf. Int.] Intercept -14.3473 4.007 -3.581 0.000 -22.224 -6.470 Year 0.7573 0.049 15.308 0.000 0.660 0.855 Weight -0.0066 0.000 -30.911 0.000 -0.007 -0.006 Omnibus: 42.504 Durbin-Watson: 1.425 Prob(Omnibus): 0.000 Jarque-Bera (JB): 71.997 Skew: 0.670 Prob(JB): 2.32e-16 Kurtosis: 4.616 Cond. No. 7.17e+04

    We can also see if introducing categorical variables helps with the regression. In this case, we only have one categorical variable, called Origin. Patsy automatically treats strings as categorical variables, so we don’t have to do anything special – but if needed we could wrap the variable name in C() to force it to be a categorical variable.

    model = ols("MPG ~ Year + Origin", data=df) results = model.fit() results.summary() OLS Regression Results Dep. Variable: MPG R-squared: 0.579 Model: OLS Adj. R-squared: 0.576 Method: Least Squares F-statistic: 178.0 Date: Sat, 20 Aug 2016 Prob (F-statistic): 1.42e-72 Time: 10:42:17 Log-Likelihood: -1191.5 No. Observations: 392 AIC: 2391. Df Residuals: 388 BIC: 2407. Df Model: 3 Covariance Type: nonrobust coef std err t P>|t| [95.0% Conf. Int.] Intercept -61.2643 5.393 -11.360 0.000 -71.868 -50.661 Origin[T.European] 7.4784 0.697 10.734 0.000 6.109 8.848 Origin[T.Japanese] 8.4262 0.671 12.564 0.000 7.108 9.745 Year 1.0755 0.071 15.102 0.000 0.935 1.216 Omnibus: 10.231 Durbin-Watson: 1.656 Prob(Omnibus): 0.006 Jarque-Bera (JB): 10.589 Skew: 0.402 Prob(JB): 0.00502 Kurtosis: 2.980 Cond. No. 1.60e+03

    You can see here that Patsy has automatically created extra variables for Origin: in this case, European and Japanese, with the ‘default’ being American. You can configure how this is done very easily – see here.

    Just for reference, you can easily get any of the statistical outputs as attributes on the results object:

    results.rsquared 0.57919459237581172 results.params Intercept -61.264305 Origin[T.European] 7.478449 Origin[T.Japanese] 8.426227 Year 1.075484 dtype: float64

    You can also really easily use the model to predict based on values you’ve got:

    results.predict({'Year':90, 'Origin':'European'}) array([ 43.00766095])

    Categories: FLOSS Project Planets
    Syndicate content