FLOSS Project Planets

FeatherCast: Mel Llaguno, Coverity Scan and ApacheCon North America

Planet Apache - Tue, 2017-05-23 07:15

ApacheCon North America 2017 attendee interview with Mel Llaguno. Mel talks to us about Coverity Scan and how it could help Apache projects.

https://feathercastapache.files.wordpress.com/2017/05/coverity-llaguno.mp3
Categories: FLOSS Project Planets

Sergey Beryozkin: Signing HTTP Attachments with Apache CXF JOSE

Planet Apache - Tue, 2017-05-23 06:17
JOSE, the primary mechanism for securing various OAuth2/OIDC tokens, slowly but surely is becoming the main technology for securing the data in the wider contexts. JOSE, alongside COSE, will become more and more visible going forward.

I talked about Apache CXF JOSE implementation in this post. One of the practical aspects of this implementation is that one can apply JOSE to securing the regular HTTP payloads, with the best attempt at keeping the streaming going made by the sender side filters, with the JOSE protection of these payloads (JWS signature or JWE encryption) being able to 'stay' with the data even beyond the HTTP request-response time if needed.

In CXF 3.1.12 I have enhanced this feature to support the signing of HTTP attachments. It depends on JWS Detached Content and Unencoded Content features which allow to integrity-protect the payload which can continue flowing to its destination in a clear form.

Combining it with the super-flexible mechanism of processing the attachments in Apache CXF, and particularly with the newly introduced Multipart filters which let pre-process individual multipart attachment streams, helped produce the final solution.  

Besides, as part of this effort, the optional binding of the outer HTTP headers to the secure JWS or JWE payloads has also been realized.

Be the first in experimenting with this IMHO very cool feature, try it and provide the feedback, enjoy !


Categories: FLOSS Project Planets

KDevelop runtimes: Docker and Flatpak integration

Planet KDE - Tue, 2017-05-23 06:10

On my last blog post I discussed about how some assumptions such as the platform developed on can affect our development. We need to minimize it by empowering the developers with good tools so that they can develop properly. To that end, I introduced runtimes in our IDE to abstract platforms (much like on Gnome’s Builder or Qt Creator).

There are different platforms that we’ll be developing for and they need to be easily reachable when coding and testing. Both switching and interacting transparently with the different platforms.

To that end I implemented 4 approaches that integrate different runtimes:

  • Docker, allows you to develop directly against virtually any system. This is especially interesting because it enables to reproduce the environment our users are having: behavior on execution and project information (i.e. the imports are the ones from the target rather the ones on our local system). Docker is a wide-spread technology in the cloud, I hope many developers will see the value in integrating the deployed environment into the IDE while they are coding.
  • Flatpak, is a solution that targets specifically desktop Linux applications. We are talking about distributing bundled applications to users, there we have the opportunity to integrate the tooling specifically to that end: from fetching dependencies to testing on other devices (see videos below).
  • Android, as you know it’s something I’ve been pushing for years. Finally we are getting to a space where the IDE can help get some set up troubles out of the way.
  • The local host, i.e. what we have now.

And remember KDevelop is extensible. Do you want snapcraft?, vagrant?, mock? Contributions are very welcome!

If there’s something better than a list of technologies and buzzwords, that’s videos. Let’s see why this could change how you develop your software.

One development, any platform

We get to develop an application and switch back and forth the target platform we are developing for.

Here I put together a short video that tests Blinken on different platforms:

One development, any device

Using the right SDK is not enough proof that the application will work as expected on every device, especially those our users will be using. Being able to easily send our application to another device to test and play around with is something I had needed for longtime. Especially important when we need to test different form factors or input devices.

In this video we can see how we can easily test an application locally and when it works just switch to Android and send to the device for proper test on the smaller touch screen.

Here we can see how we can just test an application by executing it remotely on another device. This is done by creating a bundle of the application, sending it to the device where we want to test it and executing it there.

Hassle-free contributions

You can’t deny it. You’ve wanted to fix things in the past, but you couldn’t be bothered with setting up the development environment. Both Flatpak and Docker offer the possibility to maintainers to distribute recipes to set up development platforms that can and should be integrated so that we can dedicate this 1 hour in the week-end to fixing that bug that’s been annoying us rather than reading a couple of wikis and – oh, well, never mind, gotta make dinner.

We can do this either by providing the flatpak-builder json manifest (disclaimer: the video is quite slow).

Or a Dockerfile.

You can try this today by building kdevelop git master branch, feedback is welcome. Or wait for KDevelop 5.2 later this year.

Happy hacking!

Categories: FLOSS Project Planets

Simple is Better Than Complex: How to Deploy a Django Application on RHEL 7

Planet Python - Tue, 2017-05-23 05:02

In this tutorial, you will learn how to deploy a Django application with PostgreSQL, Nginx, Gunicorn on a Red Hat Enterprise Linux (RHEL) version 7.3. For testing purpose I’m using an Amazon EC2 instance running RHEL 7.3.

Recently I had to deploy an existing Django project running on Ubuntu 16.04 to a new environment, RHEL 7.3. It gave me some headache because I don’t have much server administration skills and I wasn’t familiar with Security Enhanced Linux (SELinux) distributions, so I thought about sharing the details of the deployment so it could help someone in the same position I was.

If you are just getting started with Django deployment, and doesn’t have a good reason to be using RHEL, I suggest you use Ubuntu instead. It requires less configuration, and the process is fairly easier than using RHEL. Perhaps you could check this past tutorial: How to Deploy a Django Application to Digital Ocean.

Anyway, for this tutorial I will deploy the following Django application: github.com/sibtc/urban-train. It is just an empty Django project to demonstrate the deployment process. So, every time you see urban-train, change it for your project name.

Initial Setup

First, let’s install all the needed resources and applications. Get started by installing git, gcc and python-virtualenv. Everything should be available in the yum repository.

sudo yum -y install git gcc python-virtualenv

Create a system user for the application:

sudo groupadd --system urbantrain sudo useradd --system --gid urbantrain --shell /bin/bash --home /opt/urban-train urbantrain

Create the Django project home inside /opt:

sudo mkdir /opt/urban-train

Give the permissions to the urbantrain user:

sudo chown urbantrain:urbantrain /opt/urban-train PostgreSQL Server

Now install PostgreSQL 9.6 server and development tools:

sudo yum -y install https://yum.postgresql.org/9.6/redhat/rhel-7-x86_64/pgdg-redhat96-9.6-3.noarch.rpm sudo yum -y install postgresql96-server postgresql96-contrib postgresql96-devel

Initialize the database:

sudo /usr/pgsql-9.6/bin/postgresql96-setup initdb

Start and enable the PostgreSQL 9.6 service:

sudo systemctl start postgresql-9.6 sudo systemctl enable postgresql-9.6

Log in with the postgres user:

sudo su - postgres

Create a database user, set a password (save it for later) and create a database for the Bootcamp application:

createuser u_urban psql -c "ALTER USER u_urban WITH PASSWORD '123';" createdb --owner u_urban urban_prod

Now we have to update the authentication method of the database user in the file pg_hba.conf:

vi /var/lib/pgsql/9.6/data/pg_hba.conf

Go to the bottom of the file, find this snippet:

# TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all all peer # IPv4 local connections: host all all 127.0.0.1/32 ident # IPv6 local connections: host all all ::1/128 ident # Allow replication connections from localhost, by a user with the # replication privilege. #local replication postgres peer #host replication postgres 127.0.0.1/32 ident #host replication postgres ::1/128 ident

Change the method from ident to md5 on the IPv4 and IPv6 rows:

# TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all all peer # IPv4 local connections: host all all 127.0.0.1/32 md5 # <- here # IPv6 local connections: host all all ::1/128 md5 # <- and here # Allow replication connections from localhost, by a user with the # replication privilege. #local replication postgres peer #host replication postgres 127.0.0.1/32 ident #host replication postgres ::1/128 ident

Save the file and exit.

Now, log out from the postgres session:

exit

Restart the PostgreSQL 9.6 server:

sudo systemctl restart postgresql-9.6 Python Virtual Environment

First, log in with the urbantrain system user:

sudo su - urbantrain

Start a new python-virtualenv inside the /opt/urban-train directory:

virtualenv venv

Activate the python-virtualenv:

source venv/bin/activate

Create a directory named logs that will be used by Gunicorn and Nginx to write the logs:

mkdir logs

Clone your project’s repository inside the /opt/urban-train directory:

git clone git@github.com:sibtc/urban-train.git

Now we have to install the Python dependencies. But first, add the PostgreSQL to the path. The psycopg2 will need it to install:

export PATH=$PATH:/usr/pgsql-9.6/bin/

Upgrade the Python package manager:

pip install pip --upgrade

Install the dependencies (/opt/urban-train/urban-train/requirements.txt inside the repository):

pip install -r requirements.txt

Migrate the database:

python manage.py migrate

Collect the static assets (css, javascripts, images, etc.):

python manage.py collectstatic --noinput Gunicorn

Still logged in with the urbantrain user, let’s create a gunicorn_start file to startup the application server.

vi /opt/urban-train/gunicorn_start

Use the structure below, change the paths, user/groups etc accordingly to your environment/project:

#!/bin/bash NAME="urban_train" DJANGODIR=/opt/urban-train/urban-train USER=urban GROUP=urban WORKERS=3 BIND=unix:/opt/urban-train/run/gunicorn.sock DJANGO_SETTINGS_MODULE=urban_train.settings DJANGO_WSGI_MODULE=urban_train.wsgi LOGLEVEL=error cd $DJANGODIR source venv/bin/activate export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE export PYTHONPATH=$DJANGODIR:$PYTHONPATH exec venv/bin/gunicorn ${DJANGO_WSGI_MODULE}:application \ --name $NAME \ --workers $WORKERS \ --user=$USER \ --group=$GROUP \ --bind=$BIND \ --log-level=$LOGLEVEL \ --log-file=-

Make the gunicorn_start file executable:

chmod u+x gunicorn_start

Create a directory named run, for the unix socket file:

mkdir run Gunicorn Systemd Service

Now let’s create a systemd service file for gunicorn server to manage

First, exit the urbantrain user. Create the following systemd service file:

sudo vi /etc/systemd/system/gunicorn.service

Insert the following in the file:

[Unit] Description=gunicorn daemon After=network.target [Service] User=urbantrain Group=urbantrain WorkingDirectory=/opt/urban-train ExecStart=/opt/urban-train/gunicorn_start [Install] WantedBy=multi-user.target

Start the gunicorn systemd service we created and enable it so that it starts at boot:

sudo systemctl start gunicorn sudo systemctl enable gunicorn Nginx

The application must be served behind a proxy server. First, create a yum repo file:

sudo vi /etc/yum.repos.d/nginx.repo

Add repository information:

[nginx] name=nginx repo baseurl=http://nginx.org/packages/rhel/7/$basearch/ gpgcheck=0 enabled=1

Save and exit.

Now install nginx:

sudo yum -y install nginx

Because of the security policies of the SELinux, we need to manually add the httpd_t to the list of permissive domains, run this command:

sudo semanage permissive -a httpd_t

Now let’s create a .conf file for the project. Go to the conf.d directory:

cd /etc/nginx/conf.d/

Remove the default.conf file, and create a new one for our project:

sudo rm default.conf sudo vi urban-train.conf

Inside of the urban-train.conf file, insert the new server block:

upstream app_server { server unix:/opt/urban-train/run/gunicorn.sock fail_timeout=0; } server { listen 80; server_name IP_ADDRESS_OR_DOMAIN_NAME; # <- insert here the ip address/domain name keepalive_timeout 5; client_max_body_size 4G; access_log /opt/urban-train/logs/nginx-access.log; error_log /opt/urban-train/logs/nginx-error.log; location /static/ { alias /opt/urban-train/static/; } location /media/ { alias /opt/urban-train/media/; } location / { try_files $uri @proxy_to_app; } location @proxy_to_app { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect off; proxy_pass http://app_server; } }

Test the nginx.conf:

sudo nginx -t

Start the nginx service and enable it so that it starts at boot:

sudo systemctl start nginx sudo systemctl enable nginx Final Remarks

Everything should be working now. Do a final test, reboot the server and check if everything starts up normally:

sudo reboot

Some things that may cause trouble:

Categories: FLOSS Project Planets

S. Lott: Python under XCode?!? Cool. Sort of.

Planet Python - Tue, 2017-05-23 04:00
Dan Bader (@dbader_org)5/5/17, 10:47 PM"Running Python in Xcode: Step by Step" ericasadun.com/2016/12/04/run…

Thanks for the link. There's a fair amount of "this doesn't seem to be what Xcode was designed to do." But it does seem to work.

I'll stick with Komodo Edit.
Categories: FLOSS Project Planets

Ian Boston: Ultrasonic Antifouling

Planet Apache - Tue, 2017-05-23 03:02

The board design went off to PCBWay via web browser and 5 days later 5 boards arrived by DHL from China. The whole process was unbelievably smooth. This was the first time I had ordered boards using the output of KiCad so I was impressed with both KiCad and PCBWay. The boards were simple, being 2 layer, but complex being large with some areas needing to carry high amps. So how did I do ?

I made 1 mistake on the footprints. The 2 terminal connectors for the 600v ultrasound output didn’t have pads on both sides. This didn’t matter as being through hole the connectors soldered ok. Other than that PCBWay did exactly what I had instructed them to. Even the Arduino Mega footprint fitted perfectly.

How did it perform ?

Once populated the board initially appeared to perform well. Random frequency from 20KHz to 150KHz worked. The drive waveform from the Mostfet drivers into the Mosfet was near perfect with no high frequency ringing on the edges with levels going from 0-12v and back in much less than 1us. However I noticed some problems with the PWM control. There was none. With PWM pulses at 10% the MOSFETS would turn on for 90% of the time and drive a wildly resonant waverform through the coil. Rather like a little hammer hitting a bit pendulum and having it feedback into resonance. On further investigation the scope showed that when the Mosfet tried to switch off the inductor carried on producing a flyback voltage causing the MostFet to continue conducting till the opposing mosfet turned on. Initially I thought this was ringing, but it turned out a simple pair of 1A high frequency Schottky diodes across each winding of the primary coil returned the energy to the the 12V line eliminating the fly back. Now I had ringing, at 10MHz, but control over the power output via a digital pot. I could leave it at that, but this 10MHz would probably transmit and cause problems with other equipment on the boat.

I think the difference between the red and blue signals is due to slightly different track lengths on each Mosfet. The shorter track not ringing nearly as much shown in the blue signal. The longer track with more capacitance ringing more and inducing a parasitic ring in the blue track. To eliminate this 2 things were done. Traditional Snubber RC networks had little or no impact. So a 100nF cap as close as possible to the Drain and Source on each Mosfet (RPF50N6) eliminated some of the high frequency, and a 100uF cap on the center tap to store the energy returned to the 12V line by flyback. This reduced the peak current.

There is still some ringing, but now the frequency is less and it is less violent. The ripple on the 12V line is now less than 0.2v and filtered out by decoupling caps on the supply pins to the Ardiono Mega. All of these modifications have been accommodated on the underside of the board.

The board now produces 60W per transducer between 20 and 150 KHz at 50% PWM drawing 5A from the supply. This is very loud on my desk and far louder than the Ultrasound Antifouling installed in Isador, which seems to work. I will need to implement a control program that balances power consumption against noise levels against effectiveness, but that is all software. There are sensors on board for temperature, current and voltage so it should be possible to have the code adapt to its environment.

Board Layout mistakes

Apart from the circuit errors, I made some mistakes in the MoSFET power connections. Rev2 of the board will have the MosFETS placed as close to the primary of the transformer with identical track lengths. Hopefully this will eliminate the ringing seen on the red trace and made both line the blue trace.

I have 4 spare unpopulated PCBs. If I do a rev2 board, I will use PCBWay again. Their boards were perfect, all the mistakes were mine.

 

 

Categories: FLOSS Project Planets

Tianon Gravi: Debuerreotype

Planet Debian - Tue, 2017-05-23 02:00

Following in the footsteps of one of my favorite Debian Developers, Chris Lamb / lamby (who is quite prolific in the reproducible builds effort within Debian), I’ve started a new project based on snapshot.debian.org (time-based snapshots of the Debian archive) and some of lamby’s work for creating reproducible Debian (debootstrap) rootfs tarballs.

The project is named “Debuerreotype” as an homage to the photography roots of the word “snapshot” and the daguerreotype process which was an early method of taking photographs. The essential goal is to create “photographs” of a minimal Debian rootfs, so the name seemed appropriate (even if it’s a bit on the “mouthful” side).

The end-goal is to create and release Debian rootfs tarballs for a given point-in-time (especially for use in Docker) which should be fully reproducible, and thus improve confidence in the provenance of the Debian Docker base images.

For more information about reproducibility and why it matters, see reproducible-builds.org, which has more thorough explanations of the why and how and links to other important work such as the reproducible builds effort in Debian (for Debian package builds).

In order to verify that the tool actually works as intended, I ran builds against seven explicit architectures (amd64, arm64, armel, armhf, i386, ppc64el, s390x) and eight explicit suites (oldstable, stable, testing, unstable, wheezy, jessie, stretch, sid).

I used a timestamp value of 2017-05-16T00:00:00Z, and skipped combinations that don’t exist (such as wheezy on arm64) or aren’t supported anymore (such as wheezy on s390x). I ran the scripts repeatedly over several days, using diffoscope to compare the results.

While doing said testing, I ran across #857803, and added a workaround. There’s also a minor outstanding issue with wheezy’s reproducibility that I haven’t had a chance to dig deep very deeply into yet (but it’s pretty benign and Wheezy’s LTS support window ends 2018-05-31, so I’m not too stressed about it).

I’ve also packaged the tool for Debian, and submitted it into the NEW queue, so hopefully the FTP Masters will look favorably upon this being a tool that’s available to install from the Debian archive as well.

Categories: FLOSS Project Planets

Agiledrop.com Blog: AGILEDROP: DrupalCon session about Coding and Development

Planet Drupal - Tue, 2017-05-23 01:13
Last time, we gathered together DrupalCon Baltimore sessions about Project Management. Before that, we explored Case Studies. We promised that we will also look in some other areas. Therefore, we will this time see, which sessions were present in the area of Coding and Development. Code Standards: It's Okay to be Yourself, But Write Your Code Like Everyone Else by Alanna Burke from Chromatic In this session, attendees learned both formatting standards for their code and documentation standards, as well as some specifics for things like Twig, and object-oriented programming in Drupal 8. The… READ MORE
Categories: FLOSS Project Planets

Call to Test the Pre-Release of 5.6.0

Planet KDE - Mon, 2017-05-22 21:35

Once again a lot has been going on behind the scenes since the last release. The HTML gallery tool is back, database shrinking (e.g. purging stale thumbnails) is also supported on MySQL, grouping has been improved and additional sidecars can now be specified. Therefore the release of 5.6.0 will be (is already) delayed, as we would like to invite you to test all these features. As usual they are available in the pre-release bundles or obviously directly from the git repository. Please report any dysfunctions, unexpected behaviour or suggestions for improvement to our bug tracker.

The HTML gallery is accessible through the tools menu in the main bar of both digiKam and showFoto. It allows you to export pictures to a gallery that you can then open in any browser. There are many themes to select and you can create your own as well.

Already in 5.5.0 tests for database integrity and obsolete information have been introduced. Besides obvious data safety improvements this can free up quite a lot of space in the digiKam databases. For technical reasons only SQLite database were shrunk to this smaller size in 5.5.0. Now this is also possible for MySQL databases.

Earlier changes to the grouping behaviour proved that digiKam users have quite diverse workflows - so with the current change we try to represent that diversity.

Originally grouped items were basically hidden away. Due to requests to include grouped items in certain operations, this was changed entirely to include grouped items in (almost) all operations. Needless to say, this wasn’t such a good idea either. So now you can choose which operations should be performed on all images in a group or just the first one.
The corresponding settings lives in the configuration wizard under Miscellaneous in the Grouping tab. By default all operations are set to Ask, which will open a dialog whenever you perform this operation and grouped items are involved.

Another new capability is to recognise additional sidecars. Under the new Sidecars tab in the Metadata part of the configuration wizard you can specify any additional extension that you want digiKam to recognise as a sidecar. These files will neither be read from nor written to, but they will be moved/rename/deleted/… together with the item that they belong to.

Finally, the last important change done for the next version is to restore the geolocation bookmarks feature which do not work with bundle versions of digiKam (AppImage, MacOS, and Windows). The new bookmarker was been fully re-written and still compatible with previous geolocation bookmarks settings. It now able to display the bookmark GPS information over a map for a better usability while editing your collection.

Thanks in advance to everyone testing this new release and in general everyone using digiKam - we hope you keep enjoying this tool and spread the word!

Categories: FLOSS Project Planets

Daniel Bader: The Meaning of Underscores in Python

Planet Python - Mon, 2017-05-22 20:00
The Meaning of Underscores in Python

The various meanings and naming conventions around single and double underscores (“dunder”) in Python, how name mangling works and how it affects your own Python classes.

Single and double underscores have a meaning in Python variable and method names. Some of that meaning is merely by convention and intended as a hint to the programmer—and some of it is enforced by the Python interpreter.

If you’re wondering “What’s the meaning of single and double underscores in Python variable and method names?” I’ll do my best to get you the answer here.

In this article I’ll discuss the following five underscore patterns and naming conventions and how they affect the behavior of your Python programs:

  • Single Leading Underscore: _var
  • Single Trailing Underscore: var_
  • Double Leading Underscore: __var
  • Double Leading and Trailing Underscore: __var__
  • Single Underscore: _

At the end of the article you’ll also find a brief “cheat sheet” summary of the five different underscore naming conventions and their meaning, as well as a short video tutorial that gives you a hands-on demo of their behavior.

Let’s dive right in!

1. Single Leading Underscore: _var

When it comes to variable and method names, the single underscore prefix has a meaning by convention only. It’s a hint to the programmer—and it means what the Python community agrees it should mean, but it does not affect the behavior of your programs.

The underscore prefix is meant as a hint to another programmer that a variable or method starting with a single underscore is intended for internal use. This convention is defined in PEP 8.

This isn’t enforced by Python. Python does not have strong distinctions between “private” and “public” variables like Java does. It’s like someone put up a tiny underscore warning sign that says:

“Hey, this isn’t really meant to be a part of the public interface of this class. Best to leave it alone.”

Take a look at the following example:

class Test: def __init__(self): self.foo = 11 self._bar = 23

What’s going to happen if you instantiate this class and try to access the foo and _bar attributes defined in its __init__ constructor? Let’s find out:

>>> t = Test() >>> t.foo 11 >>> t._bar 23

You just saw that the leading single underscore in _bar did not prevent us from “reaching into” the class and accessing the value of that variable.

That’s because the single underscore prefix in Python is merely an agreed upon convention—at least when it comes to variable and method names.

However, leading underscores do impact how names get imported from modules. Imagine you had the following code in a module called my_module:

# This is my_module.py: def external_func(): return 23 def _internal_func(): return 42

Now if you use a wildcard import to import all names from the module, Python will not import names with a leading underscore (unless the module defines an __all__ list that overrides this behavior):

>>> from my_module import * >>> external_func() 23 >>> _internal_func() NameError: "name '_internal_func' is not defined"

By the way, wildcard imports should be avoided as they make it unclear which names are present in the namespace. It’s better to stick to regular imports for the sake of clarity.

Unlike wildcard imports, regular imports are not affected by the leading single underscore naming convention:

>>> import my_module >>> my_module.external_func() 23 >>> my_module._internal_func() 42

I know this might be a little confusing at this point. If you stick to the PEP 8 recommendation that wildcard imports should be avoided, then really all you need to remember is this:

Single underscores are a Python naming convention indicating a name is meant for internal use. It is generally not enforced by the Python interpreter and meant as a hint to the programmer only.

2. Single Trailing Underscore: var_

Sometimes the most fitting name for a variable is already taken by a keyword. Therefore names like class or def cannot be used as variable names in Python. In this case you can append a single underscore to break the naming conflict:

>>> def make_object(name, class): SyntaxError: "invalid syntax" >>> def make_object(name, class_): ... pass

In summary, a single trailing underscore (postfix) is used by convention to avoid naming conflicts with Python keywords. This convention is explained in PEP 8.

3. Double Leading Underscore: __var

The naming patterns we covered so far received their meaning from agreed upon conventions only. With Python class attributes (variables and methods) that start with double underscores, things are a little different.

A double underscore prefix causes the Python interpreter to rewrite the attribute name in order to avoid naming conflicts in subclasses.

This is also called name mangling—the interpreter changes the name of the variable in a way that makes it harder to create collisions when the class is extended later.

I know this sounds rather abstract. This is why I put together this little code example we can use for experimentation:

class Test: def __init__(self): self.foo = 11 self._bar = 23 self.__baz = 23

Let’s take a look at the attributes on this object using the built-in dir() function:

>>> t = Test() >>> dir(t) ['_Test__baz', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_bar', 'foo']

This gives us a list with the object’s attributes. Let’s take this list and look for our original variable names foo, _bar, and __baz—I promise you’ll notice some interesting changes.

  • The self.foo variable appears unmodified as foo in the attribute list.
  • self._bar behaves the same way—it shows up on the class as _bar. Like I said before, the leading underscore is just a convention in this case. A hint for the programmer.
  • However with self.__baz, things look a little different. When you search for __baz in that list you’ll see that there is no variable with that name.

So what happened to __baz?

If you look closely you’ll see there’s an attribute called _Test__baz on this object. This is the name mangling that the Python interpreter applies. It does this to protect the variable from getting overridden in subclasses.

Let’s create another class that extends the Test class and attempts to override its existing attributes added in the constructor:

class ExtendedTest(Test): def __init__(self): super().__init__() self.foo = 'overridden' self._bar = 'overridden' self.__baz = 'overridden'

Now what do you think the values of foo, _bar, and __baz will be on instances of this ExtendedTest class? Let’s take a look:

>>> t2 = ExtendedTest() >>> t2.foo 'overridden' >>> t2._bar 'overridden' >>> t2.__baz AttributeError: "'ExtendedTest' object has no attribute '__baz'"

Wait, why did we get that AttributeError when we tried to inspect the value of t2.__baz? Name mangling strikes again! It turns out this object doesn’t even have a __baz attribute:

>>> dir(t2) ['_ExtendedTest__baz', '_Test__baz', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_bar', 'foo', 'get_vars']

As you can see __baz got turned into _ExtendedTest__baz to prevent accidental modification:

>>> t2._ExtendedTest__baz 'overridden'

But the original _Test__baz is also still around:

>>> t2._Test__baz 42

Double underscore name mangling is fully transparent to the programmer. Take a look at the following example that will confirm this:

class ManglingTest: def __init__(self): self.__mangled = 'hello' def get_mangled(self): return self.__mangled >>> ManglingTest().get_mangled() 'hello' >>> ManglingTest().__mangled AttributeError: "'ManglingTest' object has no attribute '__mangled'"

Does name mangling also apply to method names? It sure does—name mangling affects all names that start with two underscore characters (“dunders”) in a class context:

class MangledMethod: def __method(self): return 42 def call_it(self): return self.__method() >>> MangledMethod().__method() AttributeError: "'MangledMethod' object has no attribute '__method'" >>> MangledMethod().call_it() 42

Here’s another, perhaps surprising, example of name mangling in action:

_MangledGlobal__mangled = 23 class MangledGlobal: def test(self): return __mangled >>> MangledGlobal().test() 23

In this example I declared a global variable called _MangledGlobal__mangled. Then I accessed the variable inside the context of a class named MangledGlobal. Because of name mangling I was able to reference the _MangledGlobal__mangled global variable as just __mangled inside the test() method on the class.

The Python interpreter automatically expanded the name __mangled to _MangledGlobal__mangled because it begins with two underscore characters. This demonstrated that name mangling isn’t tied to class attributes specifically. It applies to any name starting with two underscore characters used in a class context.

Now this was a lot of stuff to absorb.

To be honest with you I didn’t write these examples and explanations down off the top of my head. It took me some research and editing to do it. I’ve been using Python for years but rules and special cases like that aren’t constantly on my mind.

Sometimes the most important skills for a programmer are “pattern recognition” and knowing where to look things up. If you feel a little overwhelmed at this point, don’t worry. Take your time and play with some of the examples in this article.

Make these concepts sink in enough so that you’ll recognize the general idea of name mangling and some of the other behaviors I showed you. If you encounter them “in the wild” one day, you’ll know what to look for in the documentation.

⏰ Sidebar: What’s a “dunder” in Python?

I’ve you’ve heard some experienced Pythonistas talk about Python or watched a few conference talks you may have heard the term dunder. If you’re wondering what that is, here’s your answer:

Double underscores are often referred to as “dunders” in the Python community. The reason is that double underscores appear quite often in Python code and to avoid fatiguing their jaw muscles Pythonistas often shorten “double underscore” to “dunder.”

For example, you’d pronounce __baz as “dunder baz”. Likewise __init__ would be pronounced as “dunder init”, even though one might think it should be “dunder init dunder.” But that’s just yet another quirk in the naming convention.

It’s like a secret handshake for Python developers

Categories: FLOSS Project Planets

Aaron Morton: Bootstrapping Apache Cassandra Nodes

Planet Apache - Mon, 2017-05-22 20:00

Auto bootstrapping is a handy feature when it comes to growing an Apache Cassandra cluster. There are some unknowns about how this feature works which can lead to data inconsistencies in the cluster. In this post I will go through a bit about the history of the feature, the different knobs and levers available to operate it, and resolving some of the common issues that may arise.

Summary

Here are links to the various sections of the post to give you an idea of what I will cover.

Background

The bootstrap feature in Apache Cassandra controls the ability for the data in cluster to be automatically redistributed when a new node is inserted. The new node joining the cluster is defined as an empty node without system tables or data.

When a new node joins the cluster using the auto bootstrap feature, it will perform the following operations

  • Contact the seed nodes to learn about gossip state.
  • Transition to Up and Joining state (to indicate it is joining the cluster; represented by UJ in the nodetool status).
  • Contact the seed nodes to ensure schema agreement.
  • Calculate the tokens that it will become responsible for.
  • Stream replica data associated with the tokens it is responsible for from the former owners.
  • Transition to Up and Normal state once streaming is complete (to indicate it is now part of the cluster; represented by UN in the nodetool status).

The above operations can be seen in the logs.

Contact the seed nodes to learn about gossip state

INFO [HANDSHAKE-/127.0.0.1] 2017-05-12 16:14:45,290 OutboundTcpConnection.java:487 - Handshaking version with /127.0.0.1 INFO [GossipStage:1] 2017-05-12 16:14:45,318 Gossiper.java:1029 - Node /127.0.0.1 is now part of the cluster INFO [GossipStage:1] 2017-05-12 16:14:45,325 Gossiper.java:1029 - Node /127.0.0.2 is now part of the cluster INFO [GossipStage:1] 2017-05-12 16:14:45,326 Gossiper.java:1029 - Node /127.0.0.3 is now part of the cluster INFO [GossipStage:1] 2017-05-12 16:14:45,328 Gossiper.java:1029 - Node /127.0.0.4 is now part of the cluster INFO [SharedPool-Worker-1] 2017-05-12 16:14:45,331 Gossiper.java:993 - InetAddress /127.0.0.1 is now UP INFO [HANDSHAKE-/127.0.0.3] 2017-05-12 16:14:45,331 OutboundTcpConnection.java:487 - Handshaking version with /127.0.0.3 INFO [HANDSHAKE-/127.0.0.2] 2017-05-12 16:14:45,383 OutboundTcpConnection.java:487 - Handshaking version with /127.0.0.2 INFO [HANDSHAKE-/127.0.0.4] 2017-05-12 16:14:45,387 OutboundTcpConnection.java:487 - Handshaking version with /127.0.0.4 INFO [SharedPool-Worker-1] 2017-05-12 16:14:45,438 Gossiper.java:993 - InetAddress /127.0.0.3 is now UP INFO [SharedPool-Worker-2] 2017-05-12 16:14:45,438 Gossiper.java:993 - InetAddress /127.0.0.4 is now UP INFO [SharedPool-Worker-3] 2017-05-12 16:14:45,438 Gossiper.java:993 - InetAddress /127.0.0.2 is now UP ... INFO [main] 2017-05-12 16:14:46,289 StorageService.java:807 - Starting up server gossip

Transition to Up and Joining state

INFO [main] 2017-05-12 16:14:46,396 StorageService.java:1138 - JOINING: waiting for ring information

Contact the seed nodes to ensure schema agreement

Take note of the last entry in this log snippet.

INFO [GossipStage:1] 2017-05-12 16:14:49,081 Gossiper.java:1029 - Node /127.0.0.1 is now part of the cluster INFO [SharedPool-Worker-1] 2017-05-12 16:14:49,082 Gossiper.java:993 - InetAddress /127.0.0.1 is now UP INFO [GossipStage:1] 2017-05-12 16:14:49,095 TokenMetadata.java:414 - Updating topology for /127.0.0.1 INFO [GossipStage:1] 2017-05-12 16:14:49,096 TokenMetadata.java:414 - Updating topology for /127.0.0.1 INFO [HANDSHAKE-/127.0.0.1] 2017-05-12 16:14:49,096 OutboundTcpConnection.java:487 - Handshaking version with /127.0.0.1 INFO [GossipStage:1] 2017-05-12 16:14:49,098 Gossiper.java:1029 - Node /127.0.0.2 is now part of the cluster INFO [SharedPool-Worker-1] 2017-05-12 16:14:49,102 Gossiper.java:993 - InetAddress /127.0.0.2 is now UP INFO [GossipStage:1] 2017-05-12 16:14:49,103 TokenMetadata.java:414 - Updating topology for /127.0.0.2 INFO [HANDSHAKE-/127.0.0.2] 2017-05-12 16:14:49,104 OutboundTcpConnection.java:487 - Handshaking version with /127.0.0.2 INFO [GossipStage:1] 2017-05-12 16:14:49,104 TokenMetadata.java:414 - Updating topology for /127.0.0.2 INFO [GossipStage:1] 2017-05-12 16:14:49,106 Gossiper.java:1029 - Node /127.0.0.3 is now part of the cluster INFO [SharedPool-Worker-1] 2017-05-12 16:14:49,111 Gossiper.java:993 - InetAddress /127.0.0.3 is now UP INFO [GossipStage:1] 2017-05-12 16:14:49,112 TokenMetadata.java:414 - Updating topology for /127.0.0.3 INFO [HANDSHAKE-/127.0.0.3] 2017-05-12 16:14:49,195 OutboundTcpConnection.java:487 - Handshaking version with /127.0.0.3 INFO [GossipStage:1] 2017-05-12 16:14:49,236 TokenMetadata.java:414 - Updating topology for /127.0.0.3 INFO [GossipStage:1] 2017-05-12 16:14:49,247 Gossiper.java:1029 - Node /127.0.0.4 is now part of the cluster INFO [SharedPool-Worker-1] 2017-05-12 16:14:49,248 Gossiper.java:993 - InetAddress /127.0.0.4 is now UP INFO [InternalResponseStage:1] 2017-05-12 16:14:49,252 ColumnFamilyStore.java:905 - Enqueuing flush of schema_keyspaces: 1444 (0%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:2] 2017-05-12 16:14:49,254 Memtable.java:347 - Writing Memtable-schema_keyspaces@1493033009(0.403KiB serialized bytes, 10 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:2] 2017-05-12 16:14:49,256 Memtable.java:382 - Completed flushing .../node5/data0/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-1-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1494569684606, position=119856) INFO [InternalResponseStage:1] 2017-05-12 16:14:49,367 ColumnFamilyStore.java:905 - Enqueuing flush of schema_columnfamilies: 120419 (0%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:1] 2017-05-12 16:14:49,368 Memtable.java:347 - Writing Memtable-schema_columnfamilies@1679976057(31.173KiB serialized bytes, 541 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:1] 2017-05-12 16:14:49,396 Memtable.java:382 - Completed flushing .../node5/data0/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-tmp-ka-1-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1494569684606, position=119856) ... INFO [InternalResponseStage:5] 2017-05-12 16:14:50,824 ColumnFamilyStore.java:905 - Enqueuing flush of schema_usertypes: 160 (0%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:2] 2017-05-12 16:14:50,824 Memtable.java:347 - Writing Memtable-schema_usertypes@1946148009(0.008KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:2] 2017-05-12 16:14:50,826 Memtable.java:382 - Completed flushing .../node5/data0/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/system-schema_usertypes-tmp-ka-10-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1494569684606, position=252372) INFO [main] 2017-05-12 16:14:50,404 StorageService.java:1138 - JOINING: schema complete, ready to bootstrap

Calculate the tokens that it will become responsible for

INFO [main] 2017-05-12 16:14:50,404 StorageService.java:1138 - JOINING: waiting for pending range calculation INFO [main] 2017-05-12 16:14:50,404 StorageService.java:1138 - JOINING: calculation complete, ready to bootstrap INFO [main] 2017-05-12 16:14:50,405 StorageService.java:1138 - JOINING: getting bootstrap token

Stream replica data associated with the tokens it is responsible for from the former owners

Take note of the first and last entries in this log snippet.

INFO [main] 2017-05-12 16:15:20,440 StorageService.java:1138 - JOINING: Starting to bootstrap... INFO [main] 2017-05-12 16:15:20,461 StreamResultFuture.java:86 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Executing streaming plan for Bootstrap INFO [StreamConnectionEstablisher:1] 2017-05-12 16:15:20,462 StreamSession.java:220 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Starting streaming to /127.0.0.1 INFO [StreamConnectionEstablisher:2] 2017-05-12 16:15:20,462 StreamSession.java:220 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Starting streaming to /127.0.0.2 INFO [StreamConnectionEstablisher:3] 2017-05-12 16:15:20,462 StreamSession.java:220 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Starting streaming to /127.0.0.3 INFO [StreamConnectionEstablisher:1] 2017-05-12 16:15:20,478 StreamCoordinator.java:209 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3, ID#0] Beginning stream session with /127.0.0.1 INFO [StreamConnectionEstablisher:2] 2017-05-12 16:15:20,478 StreamCoordinator.java:209 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3, ID#0] Beginning stream session with /127.0.0.2 INFO [StreamConnectionEstablisher:3] 2017-05-12 16:15:20,478 StreamCoordinator.java:209 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3, ID#0] Beginning stream session with /127.0.0.3 INFO [STREAM-IN-/127.0.0.2] 2017-05-12 16:15:24,339 StreamResultFuture.java:166 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3 ID#0] Prepare completed. Receiving 11 files(10176549820 bytes), sending 0 files(0 bytes) INFO [STREAM-IN-/127.0.0.3] 2017-05-12 16:15:27,201 StreamResultFuture.java:180 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Session with /127.0.0.3 is complete INFO [STREAM-IN-/127.0.0.1] 2017-05-12 16:15:33,256 StreamResultFuture.java:180 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Session with /127.0.0.1 is complete INFO [StreamReceiveTask:1] 2017-05-12 16:36:31,249 StreamResultFuture.java:180 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] Session with /127.0.0.2 is complete INFO [StreamReceiveTask:1] 2017-05-12 16:36:31,256 StreamResultFuture.java:212 - [Stream #604b5690-36da-11e7-aeb6-9d89ad20c2d3] All sessions completed INFO [main] 2017-05-12 16:36:31,257 StorageService.java:1167 - Bootstrap completed! for the tokens [1577102245397509090, -713021257351906154, 5943548853755748481, -186427637333122985, 89474807595263595, -3872409873927530770, 269282297308186556, -2090619435347582830, -7442271648674805532, 1993467991047389706, 3250292341615557960, 3680244045045170206, -6121195565829299067, 2336819841643904893, 8366041580813128754, -1539294702421999531, 5559860204752248078, 4990559483982320587, -5978802488822380342, 7738662906313460122, -8543589077123834538, 8470022885937685086, 7921538168239180973, 5167628632246463806, -8217637230111416952, 7867074371397881074, -6728907721317936873, -5403440910106158938, 417632467923200524, -5024952230859509916, -2145251677903377866, 62038536271402824]

Transition to Up and Normal state once streaming is complete

INFO [main] 2017-05-12 16:36:31,348 StorageService.java:1715 - Node /127.0.0.5 state jump to NORMAL

During the bootstrapping process, the new node joining the cluster has no effect on the existing data in terms of Replication Factor (RF). However, the new node will accept new writes for the token ranges acquired while existing data from the other nodes is being streamed to it. This ensures that no new writes are missed while data changes hands. In addition, it ensures that Consistency Level (CL) is respected all the time during the streaming process and even in the case of bootstrap failure. Once the bootstrapping process for the new node completes, it will begin to serve read requests (and continue to receive writes). Like the pre-existing nodes in the cluster, it too will then have an effect on the data in terms of RF and CL.

While the bootstrapping feature can be a time saver when expanding a cluster, there are some “gotchas” that are worth noting. But before we do, we need first revisit some basics.

Back to basics

Cassandra uses a token system to work out which nodes will hold which partition keys for the primary replica of data. To work out where data is stored in the cluster, Cassandra will first apply a hashing function to the partition key. The generated hash is then used to calculate a token value using an algorithm; most commonly Murmur3 or RandomPartitioner.

As seen from the log snippets, when a new node is added to the cluster it will calculate the tokens of the different data replicas that is to be responsible for. This process where tokens are calculated and acquired by the new node is often referred to as a range movement. i.e. token ranges are being moved between nodes. Once the range movement has completed, the node will by default begin the bootstrapping process where it streams data for the acquired tokens from other nodes.

Gotchas Range movements

Whilst range movements may sound simple, the process can create implications with maintaining data consistency. A number of patches have been added over time to help maintain data consistency during range movements. A fairly well known issue was CASSANDRA-2434 where it was highlighted that range movements violated consistency for Apache Cassandra versions below 2.1.x using vnodes.

A fix was added for the issue CASSANDRA-2434 to ensure range movements between nodes were consistent when using vnodes. Prior to this patch inconsistencies could be caused during bootstrapping as per the example Jeff Jirsa gave on the dev mailing list.

Consider the case of a cluster containing three nodes A, B and D with a RF of 3. If node B was offline and a key ‘foo’ was written with CL of QUORUM, the value for key ‘foo’ would go to nodes A and D.

At a later point in time node B is resurrected and added back into the cluster. Around the same time a node C is added to the cluster and begins bootstrapping.

One of the tokens node C calculates and acquires during the bootstrap process is for key ‘foo’. Node B is the closest node with data for the newly acquired token and thus node C begins streaming from the neighbouring node B. This process violates the consistency guarantees of Cassandra. This is because the data on node C will be the same as node B, and both are missing the value for key ‘foo’.

Thus, a query with a CL of QUORUM may query nodes B and C and return no data which is incorrect, despite there being data for ‘foo’ on node A. Node D previously had the correct data, but it stopped being a replica after C was inserted into the cluster.

The above issue was solved in CASSANDRA-2434 by changing the default behaviour to always trying to perform a consistent range movement. That is, when node C is added (in the previous example), data is streamed from the correct replica it is replacing, node D. In this case all queries with CL of QUORUM for the key ‘foo’ would always return the correct value.

The JVM option cassandra.consistent.rangemovement was added as part of this patch. The option allows consistent range movements during bootstrapping to be disabled should the user desire this behaviour. This fix is no silver bullet though, because it requires that the correct node be available for a consistent range moment during a bootstrap. This may not always be possible, and in such cases there are two options:

  1. Get the required node back online (preferred option).
  2. If the required node is unrecoverable, set JVM_OPTS="$JVM_OPTS -Dcassandra.consistent.rangemovement=false" in the cassandra-env.sh file to perform inconsistent range movements when auto bootstrapping. Once bootstrapping is complete, a repair will need to be run using the following command on the node. This is to ensure the data it streamed is consistent with the rest of the replicas.
nodetool repair -full Adding multiple nodes

Another common cause of grief for users was bootstrapping multiple node simultaneously; captured in CASSANDRA-7069. Adding two new nodes simultaneously to a cluster could potentially be harmful, given the operations performed by a new node when joining. Waiting two minutes for the gossip state to propagate before adding a new node is possible, however as noted in CASSANDRA-9667, there is no coordination between nodes during token selection. For example consider that case if Node A was bootstrapped, then two minutes later Node B was bootstrapped. Node B could potentially pick token ranges already selected by Node A.

The above issue was solved in CASSANDRA-7069 by changing the default behaviour such that adding a node would fail if another node was already bootstrapping in a cluster. Similar to CASSANDRA-2434, this behaviour could be disabled by setting the JVM option JVM_OPTS="$JVM_OPTS -Dcassandra.consistent.rangemovement=false" in the cassandra-env.sh file on the bootstrapping node. This means that if cassandra.consistent.rangemovement=false is set to allow multiple nodes to bootstrap, the cluster runs the risk of violating consistency guarantees because of CASSANDRA-2434.

Changes made by CASSANDRA-7069 mean that the default behaviour forces a user to add a single node at a time to expand the cluster. This is the safest way of adding nodes to expand a cluster and ensure that the correct amount of data is streamed between nodes.

Data streaming

To further add to the confusion there is a misconception about what the auto_bootstrap property does in relation to a node being added to the cluster. Despite its name, this property controls the data streaming step only in the bootstrap process. The boolean property is by default set to true. When set to true, the data streaming step will be performed during the bootstrap process.

Setting auto_bootstrap to false when bootstrapping a new node exposes the cluster to huge inconsistencies. This is because all the other steps in the process are carried out but no data is streamed to the node. Hence, the node would be in the UN state without having any data for the token ranges it has been allocated! Furthermore, the new node without data will be serving reads and nodes that previously owned the tokens will no longer be serving reads. Effectively, the token ranges for that replica would be replaced with no data.

It is worth noting that the other danger to using auto_bootstrap set to false is no IP address collision check occurs. As per CASSANDRA-10134, if a new node has auto_bootstrap set to false and has the same address as an existing down node, the new node will take over the token range of the old node. No error is thrown, only a warning messages such as the following one below is written to the logs of the other nodes in the cluster. At the time of writing this post, the fix for this issue only appears in Apache Cassandra version 3.6 and above.

WARN [GossipStage:1] 2017-05-19 17:35:10,994 TokenMetadata.java:237 - Changing /127.0.0.3's host ID from 1938db5d-5f23-46e8-921c-edde18e9c829 to c30fbbb8-07ae-412c-baea-90865856104e

The behaviour of auto_bootstrap: false can lead to data inconsistencies in the following way. Consider the case of a cluster containing three nodes A, B and D with a RF of 3. If node B was offline and a key ‘foo’ was written with CL of QUORUM, the value for key ‘foo’ would go to nodes A and D. In this scenario Node D is the owner of the token relating to the key ‘foo’.

At a later point in time node B is resurrected and added back into the cluster. Around the same time a node C is added to the cluster with auto_bootstrap set to false and begins the joining process.

One of the tokens node C calculates and acquires during the bootstrap process is for key ‘foo’. Now node D is no longer the owner and hence its data for the key ‘foo’ will no longer be used during reads/writes. This process causes inconsistencies in Cassandra because both nodes B and C contain no data for key ‘foo’.

Thus, a query with a CL of QUORUM may query nodes B and C and return no data which is incorrect, despite there being data for ‘foo’ on node A. Node D previously had data, but it stopped being a replica after C was inserted.

This confusing behaviour is one of the reasons why if you look into the cassandra.yaml file you will notice that the auto_bootstrap configuration property is missing. Exposure of the property in the cassandra.yaml was short lived, as it was removed via CASSANDRA-2447 in version 1.0.0. As a result, the property is hidden and its default value of true means that new nodes will stream data when they join the cluster.

Adding a replacement node

So far we have examined various options that control the bootstrapping default behaviour when a new node is added to a cluster. Adding a new node is just one case where bootstrapping is performed, what about the case of replacing a node in the cluster if one goes down?

Should an existing node go down and needs to be replaced, the JVM option cassandra.replace_address can be used. Note that this option is only available for Apache Cassandra versions 2.x.x and higher. This feature has been around for a while and blogged about by other users in the past.

As the name suggests, it effectively replaces a down or dead node in the cluster with a new node. It is because of this that replace address option should only be used if the node is in a Down and Normal state (represented by DN in the nodetool status). Furthermore, there are no range movements that occur when using this feature, the new replacement node will simply inherit the old dead node’s token ranges. This is simpler than decommissioning the dead node and bootstrapping a fresh one, which would involve two range movements and two streaming phases. Yuck! To use the option, simply add JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=<IP_ADDRESS>" to the cassandra-env.sh file of the new node that will be replacing the old node. Where <IP_ADDRESS> is the IP address of the node to be replaced.

Once the node completes bootstrapping and joins the cluster, the JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=<IP_ADDRESS>" option must be removed from the cassandra-env.sh file or the node will fail to start on a restart. This is a short coming of the cassandra.replace_address feature. Many operators will typically be worried about a dead node being replaced and as a result forget to update the cassandra-env.sh file after the job is complete. It was for this reason that CASSANDRA-7356 was raised and resulted in a new option being added; cassandra.replace_address_first_boot. This option works once when Cassandra is first started and the replacement node inserted into the cluster. After that, the option is ignored for all subsequent restarts. It works in the same way as its predecessor; simply add JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address_first_boot=<IP_ADDRESS>" to the cassandra-env.sh and the new node is ready to be inserted.

Hang on! What about adding a replacement a seed node?

Ok, so you need to replace a seed node. Seed nodes are just like every other node in the cluster. As per the Apache Cassandra documentation, the only difference being seed nodes are the go to node when a new node joins the cluster.

There are a few extra steps to replace a seed node and bootstrap a new one in its place. Before adding the replacement seed node, the IP address of the seed node will need to be removed from the seed_provider list in the cassandra.yaml file and replaced with another node in the cluster. This needs to be done for all the nodes in the cluster. Naturally, a rolling restart will need to be performed for the changes to take effect. Once the change is complete the replacement node can be inserted as described in the previous section of this post.

What to do after it completes successfully

Once your node has successfully completed the bootstrapping process, it will transition to Up and Normal state (represented by UN in the nodetool status) to indicate it is now part of the cluster. At this point it is time to cleanup the nodes on your cluster. Yes, your nodes are dirty and need to be cleaned. “Why?” you ask, well the reason is the data that has been acquired by the newly added node still remains on the nodes that previously owned it. Whilst the nodes that previously owned the data have streamed it to the new node and relinquished the associated tokens, the data that was streamed still remains on the original nodes. This “orphaned” data is consuming valuable disk space, and in the cases large data sets; probably consuming a significant amount.

However, before running off to the console to remove the orphaned data from the nodes, make sure it is done as a last step in a cluster expansion. If the expansion of the cluster requires only one node to be added, perform the cleanup after the node has successfully completed bootstrapping and joined the cluster. If the expansion requires three nodes to be added, perform the cleanup after all three nodes have successfully completed bootstrapping and joined the cluster. This is because the cleanup will need to be executed on all nodes in the cluster, except for the last node that was added to the cluster. The last node added to the cluster will contain only the data it needed for the tokens acquired, where as other nodes may contain data for tokens they no longer have. It is still ok to run cleanup on the last node, it will likely return immediately after it is called.

The cleanup can be executed on each node using the following command.

nodetool cleanup -j <COMPACTION_SLOTS>

Where <COMPACTION_SLOTS> is the number of compaction slots to use for cleanup. By default this is 2. If set to 0 it will use use all available compaction threads.

It is probably worth limiting the number of compaction slots used by cleanup otherwise it could potentially block compactions.

Help! It failed

The bootstrap process for a joining node can fail. Bootstrapping will put extra load on the network so should bootstrap fail, you could try tweaking the streaming_socket_timeout_in_ms. Set streaming_socket_timeout_in_ms in the cassandra.yaml file to 24 hours (60 * 60 * 24 * 1000 = 86,400,000ms). Having a socket timeout set is crucial for catching streams that hang and reporting them via an exception in the logs as per CASSANDRA-11286.

If the bootstrap process fails in Cassandra version 2.1.x, the process will need to be restarted all over again. This can be done using the following steps.

  1. Stop Cassandra on the node.
  2. Delete all files and directories from the data, commitlog and save_cache directories but leave the directories there.
  3. Wait about two minutes.
  4. Start Cassandra on the node.

If the bootstrap process fails in Cassandra 2.2.x, the process can be easily be resumed using the following command thanks to CASSANDRA-8942.

nodetool bootstrap resume Testing the theory

We have gone through a lot of theory in this post, so I thought it would be good to test some of it out to demonstrate what can happen when bootstrapping multiple nodes at the same time.

Setup

In my test I used a three node local cluster running Apache Cassandra 2.1.14 which was created with the ccm tool. Each node was configured to use vnodes; specifically num_tokens was set to 32 in the cassandra.yaml file. The cluster was loaded with around 20 GB of data generated from the killrweather dataset. Data loading was performed in batches using cdm. Prior to starting the test the cluster looked like this.

$ ccm node1 nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 127.0.0.1 19.19 GB 32 29.1% cfb50e13-52a4-4821-bca2-4dba6061d38a rack1 UN 127.0.0.2 9.55 GB 32 37.4% 5176598f-bbab-4165-8130-e33e39017f7e rack1 UN 127.0.0.3 19.22 GB 32 33.5% d261faaf-628f-4b86-b60b-3825ed552aba rack1

It was not the most well balanced cluster, however it was good enough for testing. It should be noted that the node with IP address 127.0.0.1 was set to be the only seed node in the cluster. Taking a quick peak at the keyspace configuration in using CQLSH and we can see that it was using replication_factor: 1 i.e. RF = 1.

cqlsh> describe killrweather CREATE KEYSPACE killrweather WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true; Adding a new node

A new node (node4) was added to the cluster.

$ ccm node4 start

After a minute or so node4 was in the UJ state and began the bootstrap process.

$ ccm node1 nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 127.0.0.1 19.19 GB 32 29.1% cfb50e13-52a4-4821-bca2-4dba6061d38a rack1 UN 127.0.0.2 9.55 GB 32 37.4% 5176598f-bbab-4165-8130-e33e39017f7e rack1 UN 127.0.0.3 19.22 GB 32 33.5% d261faaf-628f-4b86-b60b-3825ed552aba rack1 UJ 127.0.0.4 14.44 KB 32 ? ae0a26a6-fab5-4cab-a189-697818be3c95 rack1

It was observed that node4 had started streaming data from node1 (IP address 127.0.0.1) and node2 (IP address 127.0.0.2).

$ ccm node4 nodetool netstats Mode: JOINING Bootstrap f4e54a00-36d9-11e7-b18e-9d89ad20c2d3 /127.0.0.1 Receiving 9 files, 10258729018 bytes total. Already received 2 files, 459059994 bytes total .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-3-Data.db 452316846/452316846 bytes(100%) received from idx:0/127.0.0.1 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-2-Data.db 6743148/6743148 bytes(100%) received from idx:0/127.0.0.1 /127.0.0.3 /127.0.0.2 Receiving 11 files, 10176549820 bytes total. Already received 1 files, 55948069 bytes total .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/127.0.0.2 Read Repair Statistics: Attempted: 0 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool Name Active Pending Completed Commands n/a 0 6 Responses n/a 0 471 Adding another new node

A few minutes later another new node (node5) was added to the cluster. To add this node to the cluster while node4 was bootstrapping the JVM option JVM_OPTS="$JVM_OPTS -Dcassandra.consistent.rangemovement=false" was added to the node’s cassandra-env.sh file. The node was then started.

$ ccm node5 start

After about a minute node5 was in the UJ state and it too began the bootstrap process.

$ ccm node1 nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 127.0.0.1 19.19 GB 32 29.1% cfb50e13-52a4-4821-bca2-4dba6061d38a rack1 UN 127.0.0.2 9.55 GB 32 37.4% 5176598f-bbab-4165-8130-e33e39017f7e rack1 UN 127.0.0.3 19.22 GB 32 33.5% d261faaf-628f-4b86-b60b-3825ed552aba rack1 UJ 127.0.0.4 106.52 KB 32 ? ae0a26a6-fab5-4cab-a189-697818be3c95 rack1 UJ 127.0.0.5 14.43 KB 32 ? a71ed178-f353-42ec-82c8-d2b03967753a rack1

It was observed that node5 had started streaming data from node2 as well; the same node that node4 was streaming data from.

$ ccm node5 nodetool netstats Mode: JOINING Bootstrap 604b5690-36da-11e7-aeb6-9d89ad20c2d3 /127.0.0.3 /127.0.0.2 Receiving 11 files, 10176549820 bytes total. Already received 1 files, 55948069 bytes total .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/127.0.0.2 /127.0.0.1 Read Repair Statistics: Attempted: 0 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool Name Active Pending Completed Commands n/a 0 8 Responses n/a 0 255

The interesting point to note when looking at the netstats was that both node4 and node5 were each streaming a Data.db file exactly 55948069 bytes from node2.

Data streaming much

It had appeared that both node4 and node5 were streaming the same data from node2. This continued as the bootstrapping process progressed; the size of the files being streamed from node2 were the same for both node4 and node5. Checking the netstats on node4 produced the following.

$ ccm node4 nodetool netstats Bootstrap f4e54a00-36d9-11e7-b18e-9d89ad20c2d3 /127.0.0.1 Receiving 9 files, 10258729018 bytes total. Already received 6 files, 10112487796 bytes total .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-13-Data.db 1788940555/1788940555 bytes(100%) received from idx:0/127.0.0.1 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-5-Data.db 7384377358/7384377358 bytes(100%) received from idx:0/127.0.0.1 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-12-Data.db 27960312/27960312 bytes(100%) received from idx:0/127.0.0.1 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-3-Data.db 452316846/452316846 bytes(100%) received from idx:0/127.0.0.1 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-11-Data.db 452149577/452149577 bytes(100%) received from idx:0/127.0.0.1 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-2-Data.db 6743148/6743148 bytes(100%) received from idx:0/127.0.0.1 /127.0.0.3 /127.0.0.2 Receiving 11 files, 10176549820 bytes total. Already received 10 files, 10162463079 bytes total .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-9-Data.db 55590043/55590043 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-6-Data.db 901588743/901588743 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-15-Data.db 14081154/14081154 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-16-Data.db 1450179/1450179 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-8-Data.db 901334951/901334951 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-10-Data.db 3622476547/3622476547 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-17-Data.db 56277615/56277615 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-4-Data.db 3651310715/3651310715 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-7-Data.db 902405063/902405063 bytes(100%) received from idx:0/127.0.0.2 Read Repair Statistics: Attempted: 0 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool Name Active Pending Completed Commands n/a 0 6 Responses n/a 0 4536

Then checking netstats on node5 produced the following.

$ ccm node5 nodetool netstats Mode: JOINING Bootstrap 604b5690-36da-11e7-aeb6-9d89ad20c2d3 /127.0.0.1 /127.0.0.3 /127.0.0.2 Receiving 11 files, 10176549820 bytes total. Already received 9 files, 10106185464 bytes total .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-2-Data.db 3651310715/3651310715 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-9-Data.db 1450179/1450179 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-3-Data.db 901588743/901588743 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-6-Data.db 55590043/55590043 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-4-Data.db 902405063/902405063 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-8-Data.db 14081154/14081154 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-5-Data.db 901334951/901334951 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-7-Data.db 3622476547/3622476547 bytes(100%) received from idx:0/127.0.0.2 Read Repair Statistics: Attempted: 0 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool Name Active Pending Completed Commands n/a 0 8 Responses n/a 0 4383

To be absolutely sure about what was being observed, I ran a command to order the netstats output by file size for both node4 and node5.

$ for file_size in $(ccm node4 nodetool netstats | grep '(100%)\ received' | grep '127.0.0.2' | tr -s ' ' | cut -d' ' -f3 | cut -d'/' -f1 | sort -g); do ccm node4 nodetool netstats | grep ${file_size} | tr -s ' '; done .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-16-Data.db 1450179/1450179 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-15-Data.db 14081154/14081154 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-9-Data.db 55590043/55590043 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-17-Data.db 56277615/56277615 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-8-Data.db 901334951/901334951 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-6-Data.db 901588743/901588743 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-7-Data.db 902405063/902405063 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-10-Data.db 3622476547/3622476547 bytes(100%) received from idx:0/127.0.0.2 .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-4-Data.db 3651310715/3651310715 bytes(100%) received from idx:0/127.0.0.2 $ for file_size in $(ccm node5 nodetool netstats | grep '(100%)\ received' | grep '127.0.0.2' | tr -s ' ' | cut -d' ' -f3 | cut -d'/' -f1 | sort -g); do ccm node5 nodetool netstats | grep ${file_size} | tr -s ' '; done .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-9-Data.db 1450179/1450179 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-8-Data.db 14081154/14081154 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-6-Data.db 55590043/55590043 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-1-Data.db 55948069/55948069 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-5-Data.db 901334951/901334951 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-3-Data.db 901588743/901588743 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-4-Data.db 902405063/902405063 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-7-Data.db 3622476547/3622476547 bytes(100%) received from idx:0/127.0.0.2 .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-tmp-ka-2-Data.db 3651310715/3651310715 bytes(100%) received from idx:0/127.0.0.2

With the exception of one file being streamed by node4, killrweather-raw_weather_data-tmp-ka-17-Data.db (size 56277615 bytes), node4 and node5 looked to be streaming the same data from node2. This was the first confirmation that node5 had stolen the tokens that where originally calculated by node4. Furthermore, it looked like node 4 was performing unnecessary streaming from node2. I noted down the file sizes displayed by node5’s netstats output to help track down data files on each node.

$ ccm node5 nodetool netstats | grep '(100%)\ received' | grep '127.0.0.2' | tr -s ' ' | cut -d' ' -f3 | cut -d'/' -f1 | sort -g > file_sizes.txt; cat file_sizes.txt 1450179 14081154 55590043 55948069 901334951 901588743 902405063 3622476547 3651310715 Token and the thief

Once both nodes had finished bootstrapping and had successfully joined the cluster it looked like this.

$ ccm node1 nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 127.0.0.1 19.19 GB 32 14.8% cfb50e13-52a4-4821-bca2-4dba6061d38a rack1 UN 127.0.0.2 9.55 GB 32 22.0% 5176598f-bbab-4165-8130-e33e39017f7e rack1 UN 127.0.0.3 19.22 GB 32 23.6% d261faaf-628f-4b86-b60b-3825ed552aba rack1 UN 127.0.0.4 19.17 GB 32 17.5% ae0a26a6-fab5-4cab-a189-697818be3c95 rack1 UN 127.0.0.5 9.55 GB 32 22.1% a71ed178-f353-42ec-82c8-d2b03967753a rack1

Using the file sizes I captured earlier from node5 netstats, I checked the data directories of node4 and node5 to confirm both nodes contained files of those sizes.

$ for file_size in $(cat file_sizes.txt); do ls -al .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/ | grep ${file_size}; done -rw-r--r-- 1 anthony staff 1450179 12 May 16:33 killrweather-raw_weather_data-ka-16-Data.db -rw-r--r-- 1 anthony staff 14081154 12 May 16:33 killrweather-raw_weather_data-ka-15-Data.db -rw-r--r-- 1 anthony staff 55590043 12 May 16:33 killrweather-raw_weather_data-ka-9-Data.db -rw-r--r-- 1 anthony staff 55948069 12 May 16:33 killrweather-raw_weather_data-ka-1-Data.db -rw-r--r-- 1 anthony staff 901334951 12 May 16:33 killrweather-raw_weather_data-ka-8-Data.db -rw-r--r-- 1 anthony staff 901588743 12 May 16:33 killrweather-raw_weather_data-ka-6-Data.db -rw-r--r-- 1 anthony staff 902405063 12 May 16:33 killrweather-raw_weather_data-ka-7-Data.db -rw-r--r-- 1 anthony staff 3622476547 12 May 16:33 killrweather-raw_weather_data-ka-10-Data.db -rw-r--r-- 1 anthony staff 3651310715 12 May 16:33 killrweather-raw_weather_data-ka-4-Data.db $ for file_size in $(cat file_sizes.txt); do ls -al .../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/ | grep ${file_size}; done -rw-r--r-- 1 anthony staff 1450179 12 May 16:36 killrweather-raw_weather_data-ka-9-Data.db -rw-r--r-- 1 anthony staff 14081154 12 May 16:36 killrweather-raw_weather_data-ka-8-Data.db -rw-r--r-- 1 anthony staff 55590043 12 May 16:36 killrweather-raw_weather_data-ka-6-Data.db -rw-r--r-- 1 anthony staff 55948069 12 May 16:36 killrweather-raw_weather_data-ka-1-Data.db -rw-r--r-- 1 anthony staff 901334951 12 May 16:36 killrweather-raw_weather_data-ka-5-Data.db -rw-r--r-- 1 anthony staff 901588743 12 May 16:36 killrweather-raw_weather_data-ka-3-Data.db -rw-r--r-- 1 anthony staff 902405063 12 May 16:36 killrweather-raw_weather_data-ka-4-Data.db -rw-r--r-- 1 anthony staff 3622476547 12 May 16:36 killrweather-raw_weather_data-ka-7-Data.db -rw-r--r-- 1 anthony staff 3651310715 12 May 16:36 killrweather-raw_weather_data-ka-2-Data.db

So both nodes contained files of the same size. I then decided to check if the files on each node that were the same size had the same data content. This check was done by performing an MD5 check of file pairs that were the same size.

$ BASE_DIR=...; DATA_DIR=data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d; for file_size in $(cat file_sizes.txt); do node_4_file=$(ls -al ${BASE_DIR}/node4/${DATA_DIR}/ | grep ${file_size} | tr -s ' ' | cut -d' ' -f9); node_5_file=$(ls -al ${BASE_DIR}/node5/${DATA_DIR}/ | grep ${file_size} | tr -s ' ' | cut -d' ' -f9); md5 ${BASE_DIR}/node4/${DATA_DIR}/${node_4_file} ${BASE_DIR}/node5/${DATA_DIR}/${node_5_file}; echo; done MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-16-Data.db) = a9edb85f70197c7f37aa021c817de2a2 MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-9-Data.db) = a9edb85f70197c7f37aa021c817de2a2 MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-15-Data.db) = 975f184ae36cbab07a9c28b032532f88 MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-8-Data.db) = 975f184ae36cbab07a9c28b032532f88 MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-9-Data.db) = f0160cf8e7555031b6e0835951e1896a MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-6-Data.db) = f0160cf8e7555031b6e0835951e1896a MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-1-Data.db) = 7789b794bb3ef24338282d4a1a960903 MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-1-Data.db) = 7789b794bb3ef24338282d4a1a960903 MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-8-Data.db) = 1738695bb6b4bd237b3592e80eb785f2 MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-5-Data.db) = 1738695bb6b4bd237b3592e80eb785f2 MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-6-Data.db) = f7d1faa5c59a26a260038d61e4983022 MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-3-Data.db) = f7d1faa5c59a26a260038d61e4983022 MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-7-Data.db) = d791179432dcdbaf9a9b315178fb04c7 MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-4-Data.db) = d791179432dcdbaf9a9b315178fb04c7 MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-10-Data.db) = 3e6623c2f06bcd3f5caeacee1917898b MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-7-Data.db) = 3e6623c2f06bcd3f5caeacee1917898b MD5 (.../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-4-Data.db) = 8775f5df08882df353427753f946bf10 MD5 (.../node5/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-2-Data.db) = 8775f5df08882df353427753f946bf10

Now I had absolute proof that both nodes did in fact stream the same data from node2. It did look as though that when node5 joined the cluster it had taken tokens calculated by node4. If this were the case, it would mean that the data files on node4 that are the same on node5 would no longer be needed. One way to prove that there is “orphaned” data on node4 i.e. data not associated to any of node4’s tokens, would be to run cleanup on the cluster. If there is orphaned data on node4 the cleanup would technically delete all or some of those files. Before running cleanup on the cluster, I took note of the files on node4 which were the same as the ones on node5.

$ for file_size in $(cat file_sizes.txt); do ls -al .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/ | grep ${file_size}; | tr -s ' ' | cut -d' ' -f9; done > node4_orphaned_files.txt; cat node4_orphaned_files.txt killrweather-raw_weather_data-ka-16-Data.db killrweather-raw_weather_data-ka-15-Data.db killrweather-raw_weather_data-ka-9-Data.db killrweather-raw_weather_data-ka-1-Data.db killrweather-raw_weather_data-ka-8-Data.db killrweather-raw_weather_data-ka-6-Data.db killrweather-raw_weather_data-ka-7-Data.db killrweather-raw_weather_data-ka-10-Data.db killrweather-raw_weather_data-ka-4-Data.db

I then ran a cleanup on all the nodes in the cluster.

$ ccm node1 nodetool cleanup $ ccm node2 nodetool cleanup $ ccm node3 nodetool cleanup $ ccm node4 nodetool cleanup $ ccm node5 nodetool cleanup $ ccm node1 nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 127.0.0.1 9.57 GB 32 14.8% cfb50e13-52a4-4821-bca2-4dba6061d38a rack1 UN 127.0.0.2 138.92 KB 32 22.0% 5176598f-bbab-4165-8130-e33e39017f7e rack1 UN 127.0.0.3 19.22 GB 32 23.6% d261faaf-628f-4b86-b60b-3825ed552aba rack1 UN 127.0.0.4 9.62 GB 32 17.5% ae0a26a6-fab5-4cab-a189-697818be3c95 rack1 UN 127.0.0.5 9.55 GB 32 22.1% a71ed178-f353-42ec-82c8-d2b03967753a rack1

From this output it was obvious that node4 contained orphaned data. Earlier I had run a nodetool status which was just after both nodes completed bootstrapping and moved to the UN state, and prior to running cleanup. The output produced at that point showed that node4 had a Load of 19.17 GB. Now after cleanup it was showing to have a load of 9.62 GB. As a final verification, I iterated through the list of files on node4 which were the same as the ones on node5 (node4_orphaned_files.txt) and checked if they still were present on node4.

$ for file_name in $(cat node4_orphaned_files.txt); do ls .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/${file_name}; done ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-16-Data.db: No such file or directory ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-15-Data.db: No such file or directory ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-9-Data.db: No such file or directory ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-1-Data.db: No such file or directory ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-8-Data.db: No such file or directory ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-6-Data.db: No such file or directory ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-7-Data.db: No such file or directory ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-10-Data.db: No such file or directory ls: .../node4/data0/killrweather/raw_weather_data-32f23d1015cb11e79d0fa90042a0802d/killrweather-raw_weather_data-ka-4-Data.db: No such file or directory

As it can be seen the files were deleted as part of the cleanup on node4. Which means that during bootstrap node4 originally calculated tokens for that data. It then asked for a list of files that related to those tokens from node2 and began streaming them. A little while later node5 was added to the cluster while node4 was still bootstrapping. It then calculated tokens that overlapped with node4’s tokens. Node5 then asked for a list of files that related to those tokens from node2 and started streaming data for them as well. The issue here is node4 was never notified that it no longer required to stream files from node2. Hence, unnecessary resources were being consumed as a result of bootstrapping two nodes at the same time.

Conclusion

Auto bootstrapping combined with vnodes is probably one of the most handy features in Cassandra. It takes the pain out of manually having to move data around ensure a continuous availability while expanding the cluster in a reliable and efficient way. There a number of knobs and levers for controlling the default behaviour of bootstrapping.

Configuration properties
  • auto_bootstrap - controls whether data is streamed to the new node when inserted.
  • streaming_socket_timeout_in_ms - sets socket timeout for streaming operations.
JVM options
  • cassandra.consistent.rangemovement - controls consistent range movements and multiple node bootstrapping.
  • cassandra.replace_address_first_boot=<IP_ADDRESS> - allows a down node to be replaced with a new node.

As demonstrated by setting the JVM option cassandra.consistent.rangemovement=false the cluster runs the risk of over streaming of data and worse still, it can violate consistency. For new users to Cassandra, the safest way to add multiple nodes into a cluster is to add them one at a time. Stay tuned as I will be following up with another post on bootstrapping.

Categories: FLOSS Project Planets

Justin Mason: Links for 2017-05-22

Planet Apache - Mon, 2017-05-22 19:58
Categories: FLOSS Project Planets

Steve Loughran: Dissent is a right: Dissent is a duty. @Dissidentbot

Planet Apache - Mon, 2017-05-22 18:52
It looks like the Russians interfered with the US elections, not just from the alleged publishing of the stolen emails, or through the alleged close links with the Trump campaign, but in the social networks, creating astroturfed campaigns and repeating the messages the country deemed important.

Now the UK is having an election. And no doubt the bots will be out. But if the Russians can do bots: so can I.

This then, is @dissidentbot.


Dissident bot is a Raspbery Pi running a 350 line ruby script tasked with heckling politicans

It offers:
  • The ability to listen to tweets from a number of sources: currently a few UK politicians
  • To respond pick a random responses from a set of replies written explicitly for each one
  • To tweet the reply after a 20-60s sleep.
  • Admin CLI over Twitter Direct Messaging
  • Live update of response sets via github.
  • Live add/remove of new targets (just follow/unfollow from the twitter UI)
  • Ability to assign a probability of replying, 0-100
  • Random response to anyone tweeting about it when that is not a reply (disabled due to issues)
  • Good PUE numbers, being powered off the USB port of the wifi base station, SSD storage and fanless naturally cooled DC. Oh, and we're generating a lot of solar right now, so zero-CO2 for half the day.
It's the first Ruby script of more than ten lines I've ever written; interesting experience, and I've now got three chapters into a copy of the Pickaxe Book I've had sitting unloved alongside "ML for the working programmer".  It's nice to be able to develop just by saving the file & reloading it in the interpreter...not done that since I was Prolog programming. Refreshing.

Without type checking its easy to ship code that's broken. I know, that's what tests are meant to find, but as this all depends on the live twitter APIs, it'd take effort, including maybe some split between Model and Control. Instead: broken the code into little methods I can run in the CLI.

As usual, the real problems surface once you go live:
  1. The bot kept failing overnight; nothing in the logs. Cause: its powered by the router and DD-WRT was set to reboot every night. Fix: disable.
  2. It's "reply to any reference which isn't a reply itself" doesn't work right. I think it's partly RT related, but not fully tracked it down.
  3. Although it can do a live update of the dissident.rb script, it's not yet restarting: I need to ssh in for that.
  4. I've been testing it by tweeting things myself, so I've been having to tweet random things during testing.
  5. Had to add handling of twitter blocking from too many API calls. Again: sleep a bit before retries.
  6. It's been blocked by the conservative party. That was because they've been tweeting 2-4 times/hour, and dissidentbot originally didn't have any jitter/sleep. After 24h of replying with 5s of their tweets, it's blocked.
The loopback code is the most annoying bug; nothing too serious though.

The DM CLI is nice, the fact that I haven't got live restart something which interferes with the workflow.
 
Because the Pi is behind the firewall, I've no off-prem SSH access.

The fact the conservatives have blocked me, that's just amusing. I'll need another account.

One of the most amusing things is people argue with the bot. Even with "bot" in the name, a profile saying "a raspberry pi", people argue.


Overall the big barrier is content.  It turns out that you don't need to do anything clever about string matching to select the right tweet: random heckles seems to blend in. That's probably a metric of political debate in social media: a 350 line ruby script tweeting random phrases from a limited set is indistinguishable from humans.

I will accept Pull Requests of new content. Also: people are free to deploy their own copies. without the self.txt file it won't reply to any random mentions, just listen to its followed accounts and reply to those with a matching file in the data dir.

If the Russians can do it, so can we.
Categories: FLOSS Project Planets

Summer of Coding!

Planet KDE - Mon, 2017-05-22 18:37
After a month of dread and panicking about the fact that Google Summer of Code results are announced in the middle of exam season... I'm happy to say I'll be doing the Rust plugin for KDevelop!Quick introMy name is Emma. Just turned 21. I'm a second-year undergrad at Imperial College London. Been programming since I was 10. I've worked on a bunch of different projects over the years. Many of them are open source. I've contributed to the KDevelop Python plugin previously. I worked at Microsoft Research in summer 2016 on the AssessMS project. I'm interested in a couple of different areas of computer science: artificial intelligence, computer vision, and lately compilers, type systems and operating systems. Favourite languages: Haskell, C++ and as of recently...RustRust is a rather new programming language, but it's already gained a lot of traction. It was voted “most loved” language by developers for the second year in a row in the StackOverflow developer survey. There have been projects made using Rust on everything from operating systems to game engines for Minecraft-like games. Despite this, IDE support is still lacking. This has to change...KDevelopKDevelop is a really great IDE, but it currently does not support Rust at all. However, it does have an easily extensible plugin architecture, so the logical conclusion is to write a Rust plugin! 
And there you have it. That was basically my reasoning when applying to KDE to do this project.What now?I had a bit of a snag with timing: my university exams were basically back to back for the past three weeks, and May is supposed to be used for community bonding, so I'm a bit behind on that. However, I have been playing around with Rust quite a bit (I started writing a small OS kernel because why not). Rust does interface quite nicely with C (aside from half of the code being littered with 'unsafe's). Still, this means my initial idea should work quite nicely. The plan is to get all necessary packages and a skeleton project set up by May 30 when coding begins.The plan for the next month: parsing Rust codeArguably the most difficult part of this whole project. Rust is, in my opinion, very similar to C++ when it comes to parsing (that is, a nightmare). So the plan is basically not to do any parsing at all. Bear with me for a moment.

The Rust compiler is nicely split up into different modules. One of those is the syntax parsing library, appropriately named libsyntax. Normally, it's private, except in the Rust nightly compiler (mainly for debugging purposes I suppose). However, a fork of it is available for the stable branch, named the syntex_syntax package. Several other Rust tools including rustfmt, Rust Racer and Rust Language Server use this package, so I'll assume it's stable.

It does the parsing and provides a nice visitor-pattern approach to traversing the AST. Hook this up to C++ with some foreign function calls and that's about it for parsing.

Semantic highlighting at this point becomes a matter of traversing the AST produced by the syntax parsing library (which includes the ranges of all elements), and constructing the appropriate structures on KDevelop's side.
And afterwards...The final goal is to have full language support, which includes semantic highlighting, navigation, code completion, as many possible code refactoring/generation options as possible, and debugging. Some of these partially overlap: highlighting, navigation and completions all depend on building a knowledge base of the code. Debugging, should be a matter of hooking up GDB/LLDB, which KDevelop already supports, for Rust-compiled objects. Finally, refactoring and code generation should be quite fun to do, and I think that would make KDevelop the Rust IDE.
Stay tuned for updates...
Categories: FLOSS Project Planets

NumFOCUS: NumFOCUS Awards Small Development Grants to Projects

Planet Python - Mon, 2017-05-22 17:52
This spring the NumFOCUS Board of Directors awarded targeted small development grants to applicants from or approved by our sponsored and affiliated projects. In the wake of a successful 2016 end-of-year fundraising drive, NumFOCUS wanted to direct the donated funds to our projects in a way that would have impact and visibility to donors and […]
Categories: FLOSS Project Planets

Kate 17.04.1 available for Windows

Planet KDE - Mon, 2017-05-22 15:03

Installers for Kate 17.04.1 are now available for download!

This release includes, besides bug-fixing and features, an update to the search in files plugin. The search-while-you-type in the current file should not “destroy” your last search in files results as easily as previously. The search-combo-box-history handling is also improved.

Grab it now at download.kde.org:  Kate-setup-17.04.1-KF5.34-32bit or Kate-setup-17.04.1-KF5.34-64bit

Categories: FLOSS Project Planets

Gunnar Wolf: Open Source Symposium 2017

Planet Debian - Mon, 2017-05-22 13:21

I travelled (for three days only!) to Argentina, to be a part of the Open Source Symposium 2017, a co-located event of the International Conference on Software Engineering.

This is, all in all, an interesting although small conference — We are around 30 people in the room. This is a quite unusual conference for me, as this is among the first "formal" academic conference I am part of. Sessions have so far been quite interesting.
What am I linking to from this image? Of course, the proceedings! They managed to publish the proceedings via the "formal" academic channels (a nice hard-cover Springer volume) under an Open Access license (which is sadly not usual, and is unbelievably expensive). So, you can download the full proceedings, or article by article, in EPUB or in PDF...
...Which is very very nice :)
Previous editions of this symposium have also their respective proceedings available, but AFAICT they have not been downloadable.
So, get the book; it provides very interesant and original insights into our community seen from several quite novel angles!

AttachmentSize oss2017_cover.png84.47 KB
Categories: FLOSS Project Planets

Drupal Modules: The One Percent: Drupal Modules: The One Percent — Footermap (video tutorial)

Planet Drupal - Mon, 2017-05-22 12:54
Drupal Modules: The One Percent — Footermap (video tutorial) NonProfit Mon, 05/22/2017 - 11:54 Episode 28

Here is where we bring awareness to Drupal modules running on less than 1% of reporting sites. Today we'll investigate Footermap, a module which renders the results expanded menus in a block.

Categories: FLOSS Project Planets

Valuebound: How to set the right expectations for project delivery?

Planet Drupal - Mon, 2017-05-22 12:46

Setting a clear list of expectation to the client for a project delivery goes a long way to great client relationships. A mismatched and misunderstood project goal and target always leads to dissatisfaction among team members, account head, and all other stakeholders.

I manage a team of a few developers who build web applications in Drupal. While working on projects with my team, I have had the chance to practice a few of the points that I have mentioned in the article. It has not only kept us on track but also kept people happy and motivated.

What should you do? Be involved from the beginning

When you begin a project makes sure that you and your team members are involved in the project from the beginning. There are times when the team would expand…

Categories: FLOSS Project Planets

Colm O hEigeartaigh: Security advisories issued for Apache CXF Fediz

Planet Apache - Mon, 2017-05-22 12:23
Two security advisories were recently issued for Apache CXF Fediz. In addition to fixing these issues, the recent releases of Fediz impose tighter security constraints in some areas by default compared to older releases. In this post I will document the advisories and the other security-related changes in the recent Fediz releases.

1) Security Advisories

The first security advisory is CVE-2017-7661: "The Apache CXF Fediz Jetty and Spring plugins are vulnerable to CSRF attacks.". Essentially, both the Jetty 8/9 and Spring Security 2/3 plugins are subject to a CSRF-style vulnerability when the user doesn't complete the authentication process. In addition, the Jetty plugins are vulnerable even if the user does first complete the authentication process, but only the root context is available as part of this attack.

The second advisory is CVE-2017-7662: "The Apache CXF Fediz OIDC Client Registration Service is vulnerable to CSRF attacks". The OIDC client registration service is a simple web application that allows the creation of clients for OpenId Connect, as well as a number of other administrative tasks. It is vulnerable to CSRF attacks, where a malicious application could take advantage of an existing session to make changes to the OpenId Connect clients that are stored in the IdP.

2) Fediz IdP security constraints

This section only concerns the WS-Federation (and SAML-SSO) IdP in Fediz. The WS-Federation RP application sends its address via the 'wreply' parameter to the IdP. For SAML SSO, the address to reply to is taken from the consumer service URL of the SAML SSO Request. Previously, the Apache CXF Fediz IdP contained an optional 'passiveRequestorEndpointConstraint' configuration value in the 'ApplicationEntity', which allows the admin to specify a regular expression constraint on the 'wreply' URL.

From Fediz 1.4.0, 1.3.2 and 1.2.4, a new configuration option is available in the 'ApplicationEntity' called 'passiveRequestorEndpoint'. If specified, this is directly matched against the 'wreply' parameter. In a change that breaks backwards compatibility, but that is necessary for security reasons, one of 'passiveRequestorEndpointConstraint' or 'passiveRequestorEndpoint must be specified in the 'ApplicationEntity' configuration. This ensures that the user cannot be redirected to a malicious client. Similarly, new configuration options are available called 'logoutEndpoint' and 'logoutEndpointConstraint' which validate the 'wreply' parameter in the case of redirecting the user after logging out, one of which must be specified.

3) Fediz RP security constraints

This section only concerns the WS-Federation RP plugins available in Fediz. When the user tries to log out of the Fediz RP application, a 'wreply' parameter can be specified to give the address that the Fediz IdP can redirect to after logout is complete. The old functionality was that if 'wreply' was not specified, then the RP plugin instead used the value from the 'logoutRedirectTo' configuration parameter.

From Fediz 1.4.0, 1.3.2 and 1.2.4, a new configuration option is available called 'logoutRedirectToConstraint'. If a 'wreply' parameter is presented, then it must match the regular expression that is specified for 'logoutRedirectToConstraint', otherwise the 'wreply' value is ignored and it falls back to 'logoutRedirectTo'. 
Categories: FLOSS Project Planets
Syndicate content