Colm O hEigeartaigh: Securing Apache Solr - part I

Planet Apache - Mon, 2017-06-26 05:46
This is the first post in a series of articles on securing Apache Solr. In this post we will look at deploying an example SolrCloud instance and securing access to it via basic authentication.

1) Install and deploy a SolrCloud example

Download and extract Apache Solr (6.6.0 was used for the purpose of this tutorial). Now start SolrCloud via:
  • bin/solr -e cloud
Accept all of the default options. This creates a cluster of two nodes, with a collection "gettingstarted" split into two shards and two replicas per-shard. A web interface is available after startup at: http://localhost:8983/solr/.

Once the cluster is up and running we can post some data to the collection we have created via the REST interface:
  • curl http://localhost:8983/solr/gettingstarted/update -d '[ {"id" : "book1", "title_t" : "The Merchant of Venice", "author_s" : "William Shakespeare"}]'
  • curl http://localhost:8983/solr/gettingstarted/update -d '[ {"id" : "book2", "title_t" : "Macbeth", "author_s" : "William Shakespeare"}]'
  • curl http://localhost:8983/solr/gettingstarted/update -d '[ {"id" : "book3", "title_t" : "Death of a Salesman", "author_s" : "Arthur Miller"}]'
We can search the REST interface to for example return all entries by William Shakespeare as follows:
  • curl http://localhost:8983/solr/gettingstarted/query?q=author_s:William+Shakespeare
2) Authenticating users to our SolrCloud instance

Now that our SolrCloud instance is up and running, let's look at how we can secure access to it, by using HTTP Basic Authentication to authenticate our REST requests. Download the following security configuration which enables Basic Authentication in Solr:
Two users are defined - "alice" and "bob" - both with password "SolrRocks". Now upload this configuration to the Apache Zookeeper instance that is running with Solr:
  • server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:9983 -cmd putfile /security.json security.json
Now try to run the search query above again using Curl. A 401 error will be returned. Once we specify the correct credentials then the request will work as expected, e.g.:
  • curl -u alice:SolrRocks http://localhost:8983/solr/gettingstarted/query?q=author_s:Arthur+Miller
Categories: FLOSS Project Planets

heykarthikwithu: Drupal 8 - Introduction to Plugin System and how its awesome in Drupal 8?

Planet Drupal - Mon, 2017-06-26 03:22
Drupal 8 - Introduction to Plugin System and how its awesome in Drupal 8?

Drupal 8, Introduction to Plugin System, and how plugin system is awesome in drupal 8?

heykarthikwithu Monday, 26 June 2017 - 12:52:01 - IST, Asia/Kolkata
Categories: FLOSS Project Planets

First coding month

Planet KDE - Sun, 2017-06-25 23:46
Heading towards the first evaluation:

Even the exams and graduation project work this month, heading towards the first evaluation according to timeline with the planned UI ready for further development and started the search with examining a similar tool in gimp, and had more insights about the details of how the “Image Editor” works on a deeper level and how I should work to get to my next milestone of static parts cloning supporting the variable radius within the next 11 days.

Categories: FLOSS Project Planets

Alex Clark: SAML: 1, Alex Clark: 0

Planet Python - Sun, 2017-06-25 20:00

Update: OOPS looks like this one got away from me (got published prematurely). I meant to write about Django & SAML about a month ago. Here is that post finally.

Courtesy of quickmeme


I don’t like to give up on a technical challenge, particularly when the progress is slow-but-consistent. It’s only when I know I can’t make any discernible progress easily (or at all) that I can force myself to give up. I’m not talking about giving up for the day and getting back to it. I’m talking about declaring failure and getting back to it WAY later, if at all. Well, this post is about how SAML got the better of me about one month ago. I really wanted this particular task to be successful, but I eventually had to give up and declare the task “too hard for me given the context, and possibly not even a good idea in the first place”.

Because I don’t explicitly mention this anywhere else in the article, I’ll mention it now: SAML is used to implement “single sign on”. (Cue debate on merits of this approach vs. other approaches e.g. OAuth2.)

The task

The task was to make Django act as a SAML Identity Provider (IdP), enough to convince a commercial service, acting as a SAML Service Provider (SP), to use it as one. While I didn’t achieve the desired end result, a learned a ton of information about SAML that I never really wanted to know, but feel better and good about now that I know it.

IdP vs SP

I’m not going to go into too much detail here but for the unfamiliar, the SAML protocol defines both Identity Providers (IdP) and Service Providers (SP). SAML Identity Providers answer requests from SAML Service Providers and make the decision about whether the principal (e.g. user) is allowed to access the Service Provider.

After discovering this fact, confusion about this aspect of the SAML protocol lingered and continues to linger. Does that mean the IdP is or contains the user database? As of right now, I don’t think so. But that wasn’t clear initially (and still isn’t, but perhaps is now “less unclear”).

No, the IdP is not the user database. Rather, the IdP is “the thing that answers SAML requests in accordance with the protocol’s definition of how to do so.” But it can’t hurt if the IdP needs to go no further than Django’s user database to validate an SP’s SAML assertion, right? I certainly thought so initially, and spent a whole lot of time trying to make it so.


Turns out implementing SAML presents many challenges, one of which is one must “deal with” XML. I don’t have any specific gripes about XML, but parsing XML does add a layer of complexity I could have done without.


You know what’s worse than parsing XML? Signing your XML. And you know what’s worse than that? Signing only a portion of it. (Lingering bitter sentiments from the me-of-one-month-ago.)


First, I spent some time researching the existing available options. I took it as a “red flag” not a lot of folks seemed to be using Django as an IdP, but I thought worst case, I’d be creating an insecure-but-viable-proof-of-concept. And I got close, but lost patience when I couldn’t get the commercial Service Provider to validate my SAML assertions (oh, and let’s not forget my confusion about SP-initiated vs IdP-initiated logins; I eventually decided that the “flow” of IdP-initiated was simpler and focused mostly on that; SP -> IdP -> SP, vs IdP -> SP.)

Eventually I created five, count them five, separate repositories while tackling SAML. Part of that proliferation-of-repositories is possibly due to some anti-branching sentiment on my part (I’m not anti-branching, but sometimes I prefer to “start clean” instead of branching):


I found a bunch of cool Python tooling for dealing with all-things-SAML that I will make a note of here:

TL;DR: Consider a commercial IdP like OneLogin or Auth0.

Categories: FLOSS Project Planets

Andreas Bombe: PDP-8/e Replicated — Introduction

Planet Debian - Sun, 2017-06-25 14:58

I am creating a replica of the DEC PDP-8/e architecture in an FPGA from schematics of the original hardware. So how did I end up with a project like this?

The story begins with me wanting to have a computer with one of those front panels that have many, many lights where you can really see, in real time, what the computer is doing while it is executing code. Not because I am nostalgic for a prior experience with any of those — I was born a bit too late for that and my first computer as a kid was a Commodore 64.

Now, the front panel era ended around 40 years ago with the advent of microprocessors and computers of that age and older that are complete and working are hard to find and not cheap. And even if you do, there’s the issue of weight, size (complete systems with peripherals fill at least a rack) and power consumption. So what to do — build myself a small one with modern technology of course.

While there’s many computer architectures of that era to choose from, the various PDP machines by DEC are significant and well known (and documented) due to their large numbers. The most important are probably the 12 bit PDP-8, the 16 bit PDP-11 and the 36 bit PDP-10. While the PDP-11 is enticing because of the possibility to run UNIX I wanted to start with something simpler, so I chose the PDP-8.

My implementation on display next to a real PDP-8/e at VCFe 18.0

The Original

DEC started the PDP-8 line of computers programmed data processors designed as low cost machines in 1965. It is a quite minimalist 12 bit architecture based on the earlier PDP-5, and by minimalist I mean seriously minimal. If you are familiar with early 8 bit microprocessors like the 6502 or 8080 you will find them luxuriously equipped in comparison.

The PDP-8 base architecture has a program counter (PC) and an accumulator (AC)1. That’s it. There are no pointer or index registers2. There is no stack. It has addition and AND instructions but subtractions and OR operations have to be manually coded. The optional Extended Arithmetic Element adds the MQ register but that’s really it for visible registers. The Wikipedia page on the PDP-8 has a good detailed description.

Regarding technology, the PDP-8 series has been in production long enough to get the whole range of implementations from discrete transistor logic to microprocessors. The 8/e which I target was right in the middle, implemented in TTL logic where each IC contains multiple logic elements. This allowed the CPU itself (including timing generator) to fit on three large circuit boards plugged into a backplane. Complete systems would have at least another board for the front panel and multiple boards for the core memory, then additional boards for whatever options and peripherals were desired.

Design Choices and Comparisons

I’m not the only one who had the idea to build something like that, of course. Among the other modern PDP-8 implementations with a front panel, probably the most prominent project is the Spare Time Gizmos SBC6120 which is a PDP-8 single board computer built around the Harris/Intersil HD-6120 microprocessor, which implementes the PDP-8 architecture, combined with a nice front panel. Another is the PiDP-8/I, which is another nice front panel (modeled after the 8/i which has even more lights) driven by the simh simulator running under Linux on a Raspberry Pi.

My goal is to get front panel lights that appear exactly like the real ones in operation. This necessitates driving the lights at full speed as they change with every instruction or even within instructions for some display selections. For example, if you run a tight loop that does nothing but increment AC while displaying that register, it would appear that all lights are lit at equal but less than full brightness. The reason is that the loop runs at such a high speed that even the most significant bit, which is blinking the slowest, is too fast to see flicker. Hence they are all effectively 50% on, just at different frequencies, and appear to be constantly light at the same brightness.

This is where the other projects lack what I am looking for. The PiDP-8/I is a multiplexed display which updates at something like 30 Hz or 60 Hz, taking whatever value is current in the simulation software at the time. All the states the lights took inbetween are lost and consequently there is flickering where there shouldn’t be. On the SBC6120 at least the address lines appear to update at full speed as these are the actual RAM address lines. However the used 6120 microprocessor does not have required data for the indicator display externally available. Instead, the SBC6120 runs an interrupt at 30 Hz to trap into its firmware/monitor program which then reads the current state and writes it to the front panel display, which is essentially just another peripheral. A different considerable problem with the SBC6120 is its use of the 6100 microprocessor family ICs, which are themselves long out of production and not trivial (or cheaply) to come by.

Given that the way to go is to drive all lights in step with every cycle3, this can be done by a software running on a dedicated microcontroller — which is how I started — or by implementing a real CPU with all the needed outputs in an FPGA — which is the project I am writing about.

In the next post I will give on overview of the hardware I built so far and some of the features that are yet to be implemented.

  1. With an associated link bit which is a little different from a carry bit in that it is treated as a thirteenth bit, i.e. it will be flipped rather than set when a carry occurs. [return]
  2. Although there are 8 specially treated memory addresses that will pre-increment when used in indirect addressing. [return]
  3. Basic cycles on the PDP-8/e are 1.4 µs for memory modifying cycles and fast cycles of 1.2 µs for everything else. Instructions can be one to three cycles long. [return]
Categories: FLOSS Project Planets

Steinar H. Gunderson: Frame queue management in Nageru 1.6.1

Planet Debian - Sun, 2017-06-25 12:43

Nageru 1.6.1 is on its way, and what was intended to only be a release centered around monitoring improvements (more specifically a full set of native Prometheus] metrics) actually ended up getting a fairly substantial change to how Nageru manages its frame queues. To understand what's changing and why, it's useful to first understand the history of Nageru's queue management. Nageru 1.0.0 started out with a fairly simple scheme, but with some basics that are still relevant today: One of the input cards was deemed the master card, and whenever it delivers a frame, the master clock ticks and an output frame is produced. (There are some subtleties about dropped frames and/or the master card changing frame rates, but I'm going to ignore them, since they're not important to the discussion.)

To this end, every card keeps a preallocated frame queue; when a card delivers a frame, it's put into the queue, and when the master clock ticks, it tries picking out one frame from each of the other card's queues to mix together. Note that “mix” here could be as simple as picking one input and throwing all the other ones away; the queueing algorithm doesn't care, it just feeds all of them to the theme and lets that run whatever GPU code it needs to match the user's preferences.

The only thing that really keeps the queues bounded is that the frames in them are preallocated (in GPU memory), so if one queue gets longer than 16 frames, Nageru starts dropping it. But is 16 the right number? There are two conflicting demands here, ignoring memory usage:

  • You want to keep the latency down.
  • You don't want to run out of frames in the queue if you can avoid it; if you drop too aggressively, you could find yourself at the next frame with nothing in the queue, because the input card hasn't delivered it yet when the master card ticks. (You could argue one should delay the output in this case, but for how long? And if you're using HDMI/SDI output, you have no such luxury.)

The 1.0.0 scheme does about as well as one could possibly hope in never dropping frames, but unfortunately, it can be pretty poor at latency. For instance, if your master card runs at 50 Hz and you have a 60 Hz card, the latter will eventually build up a delay of 16 * 16.7 ms = 266.7 ms—clearly unacceptable, and rather unneeded.

You could ask the user to specify a queue length, but the user probably doesn't know, and also shouldn't really have to care—more knobs to twiddle are a bad thing, and even more so knobs the user is expected to twiddle. Thus, Nageru 1.2.0 introduced queue autotuning; it keeps a running estimate on how big the queue needs to be to avoid underruns, simply based on experience. If we've been dropping frames on a queue and then there's an underrun, the “safe queue length” is increased by one, and if the queue has been having excess frames for more than a thousand successive master clock ticks, we reduce it by one again. Whenever the queue has more than this “safe” number, we drop frames.

This was simple, effective and largely fixed the problem. However, when adding metrics, I noticed a peculiar effect: Not all of my devices have equally good clocks. In particular, when setting up for 1080p50, my output card's internal clock (which assumes the role of the master clock when using HDMI/SDI output) seems to tick at about 49.9998 Hz, and my simple home camcorder delivers frames at about 49.9995 Hz. Over the course of an hour, this means it produces one more frame than you should have… which should of course be dropped. Having an SDI setup with synchronized clocks (blackburst/tri-level) would of course fix this problem, but most people are not so lucky with their cameras, not to mention the price of PC graphics cards with SDI outputs!

However, this happens very slowly, which means that for a significant amount of time, the two clocks will very nearly be in sync, and thus racing. Who ticks first is determined largely by luck in the jitter (normal is maybe 1ms, but occasionally, you'll see delayed delivery of as much as 10 ms), and this means that the “1000 frames” estimate is likely to be thrown off, and the result is hundreds of dropped frames and underruns in that period. Once the clocks have diverged enough again, you're off the hook, but again, this isn't a good place to be.

Thus, Nageru 1.6.1 change the algorithm around yet again, by incorporating more data to build an explicit jitter model. 1.5.0 was already timestamping each frame to be able to measure end-to-end latency precisely (now also exposed in Prometheus metrics), but from 1.6.1, they are actually used in the queueing algorithm. I ran several eight- to twelve-hour tests and simply stored all the event arrivals to a file, and then simulated a few different algorithms (including the old algorithm) to see how they fared in measures such as latency and number of drops/underruns.

I won't go into the full details of the new queueing algorithm (see the commit if you're interested), but the gist is: Based on the last 5000 frames, it tries to estimate the maximum possible jitter for each input (ie., how late the frame could possibly be). Based on this as well as clock offsets, it determines whether it's really sure that there will be an input frame available on the next master tick even if it drops the queue, and then trims the queue to fit.

The result is pretty satisfying; here's the end-to-end latency of my camera being sent through to the SDI output:

As you can see, the latency goes up, up, up until Nageru figures it's now safe to drop a frame, and then does it in one clean drop event; no more hundreds on drops involved. There are very late frame arrivals involved in this run—two extra frame drops, to be precise—but the algorithm simply determines immediately that they are outliers, and drops them without letting them linger in the queue. (Immediate dropping is usually preferred to sticking around for a bit and then dropping it later, as it means you only get one disturbance event in your stream as opposed to two. Of course, you can only do it if you're reasonably sure it won't lead to more underruns later.)

Nageru 1.6.1 will ship before Solskogen, as I intend to run it there :-) And there will probably be lovely premade Grafana dashboards from the Prometheus data. Although it would have been a lot nicer if Grafana were more packaging-friendly, so I could pick it up from stock Debian and run it on armhf. Hrmf. :-)

Categories: FLOSS Project Planets

Go support in KDevelop. GSoC week 4. DU-Chain time!

Planet KDE - Sun, 2017-06-25 12:24
During this week, I decided to spend more time on language support: code completion, highlighting and so on. This part is provided by DU-Chain. Du-Chain stands for Definition-Use chain which consist of various contexts, declarations in these contexts and usages of these declarations.

First change improved declaration of variables in parameters of anonymous functions. In Go language, it is possible to define anonymous function and assign it to variable, pass as parameter (people use it for various callbacks for example) or simple call it. Before my change, parameters of anonymous functions were treated as declarations only in case of assigning that function to variable. Thus, if, for example, you typed the example of Gin web framework usage:
package main
import "gopkg.in/gin-gonic/gin.v1"
func main() {
r := gin.Default()
r.GET("/ping", func(c *gin.Context) {
c.JSON(200, gin.H{
"message": "pong",
r.Run() // listen and serve on
}you would end up with “c” not being highlighted / treated as variable. After my change, parameters of anonymous functions are treated as variable declarations in all three cases: assigning, passing and calling (see screenshots).

Second change is under review and is aimed at adding code completion from embedded structs. In Go language, there is no such thing as inheritance – composition is preferred over it. Composition often has drawback – we need to pass all calls to “base” methods so there will be a lot of boilerplate code. In Go this problem is solved using “embedding” structs so their fields and methods are added to top-level struct. For example, struct A has a method Work and struct B embeds struct A. So both B.Work() and B.A.Work() are correct. Because of that we need to travel over all embedding tree for retrieving all possible completions – this is what my second change is aimed at.

Third change added errors of parsing as problems in “Problems” view.

Fourth change fixed misplacing usage information in wrong context. Before that change, information of variable usages was placed in top context, which had led to not working semantic highlighting – variable declarations had different colors but usages were all same. Therefore, while improving overall correctness of generated DU-Chain I also fixed that issue and now variable usages are colored too (see screenshots)!
Before After

Apart from DU-Chain improvements I got merged a basic project manager plugin which offers a template of simple console Go application and allows to build Go projects easier.

Looking forward to next week!
Categories: FLOSS Project Planets

That’s one small step for a man, one giant leap for GSoC project.

Planet KDE - Sun, 2017-06-25 11:33

For the last month the main time I took the exams, because of this I did not do much for my project. Nevertheless, I implemented the basic primitives and tested them.

Let me tell you about them.

Wet map.

Water is the main part in watercolors. That’s why I started with this.

Wet map contains 2 types of information: water value and speed vector. If the first parameter is clear, then the second one needs explanations. Speed vector needs for rewetting our splats (take it in mind, I will explain what this later).

All this values Wet map contains in KisPaintDevice:

KisPaintDeviceSP m_wetMap;

As the color space was chosen rgb16:

m_wetMap = new KisPaintDevice(KoColorSpaceRegistry::instance()->rgb16());

And there are information about water value and speed vector in pixel data.

But in this form Paint Device can’t visualize wet map correctly:

So I transorm Paint Device for visualizing wet map correctly (because it will be important for artists, I think). And now it looks like this:


My implementation is based on a procedural brush. Every brush stamp is a union of dynamic splats. Here you can see the behavior of splats:

Also I tested reweting (when splat go to flowing state from fixed state):

And as a final test I made splat generator for simulating strokes:

What next?

It’s high time to get splats to work in Krita. So I’m going to finish my plugin, and test splats behavior. But it will be primitive:

  1. Clear canvas for updating splats
  2. No undo/redo
  3. Stupid singleton for splat storage

Categories: FLOSS Project Planets

Lars Wirzenius: Obnam 1.22 released (backup application)

Planet Debian - Sun, 2017-06-25 08:41

I've just released version 1.22 of Obnam, my backup application. It is the first release for this year. Packages are available on code.liw.fi/debian and in Debian unstable, and source is in git. A summary of the user-visible changes is below.

For those interested in living dangerously and accidentally on purpose deleting all their data, the link below shows that status and roadmap for FORMAT GREEN ALBATROSS. http://distix.obnam.org/obnam-dev/182bd772889544d5867e1a0ce4e76652.html

Version 1.22, released 2017-06-25
  • Lars Wirzenius made Obnam log the full text of an Obnam exception/error message with more than one line. In particular this applies to encryption error messages, which now log the gpg output.

  • Lars Wirzenius made obnam restore require absolute paths for files to be restored.

  • Lars Wirzenius made obnam forget use a little less memory. The amount depends on the number of genrations and the chunks they refer to.

  • Jan Niggemann updated the German translation of the Obnam manual to match recent changes in the English version.

  • SanskritFritz and Ian Cambell fixed the kdirstat plugin.

  • Lars Wirzenius changed Obnam to hide a Python stack trace when there's a problem with the SSH connection (e.g., failure to authenticate, or existing connection breaks).

  • Lars Wirzenius made the Green Albatross version of obnam forget actually free chunks that are no longer used.

Categories: FLOSS Project Planets

The achievements during the first GSoC period in KStars

Planet KDE - Sun, 2017-06-25 03:56

I was able to do major improvements to the build system of KStars. I think more and more open-source projects should pick up these low-hanging fruits with CMake and Clang:

- CCache: Speed-up the development time and working with git branches by caching the compiled object files.
- Unity Build: This simple "hack" can reduce the build time dramatically by compiling temporary C++ meta files in which the normal C++ files are included. The build time can be speeded up to 2x-3x for bigger projects.
- Clang Sanitizers: Use undefined behavior and address sanitizers to hunt down memory handling errors. Although the executable must be recompiled with Clang and special compiler flags, the resulted binary will run with minimal slowdown. It is not a complete replacement, but these sanitizers can catch most of the problems found by Valgrind during normal runtime.
- Clang Format: Format the source code with a real compiler engine.

More details are on our wiki page:

Categories: FLOSS Project Planets

Shirish Agarwal: Dreams don’t cost a penny, mumma’s boy :)

Planet Debian - Sun, 2017-06-25 00:52

This one I promise will be short

Categories: FLOSS Project Planets

Justin Mason: Links for 2017-06-24

Planet Apache - Sat, 2017-06-24 19:58
Categories: FLOSS Project Planets

Lisandro Damián Nicanor Pérez Meyer: Qt 5.7 submodules that didn't make it to Stretch but will be in testing

Planet Debian - Sat, 2017-06-24 18:41
There are two Qt 5.7 submodules that we could not package in time for Strech but are/will be available in their 5.7 versions in testing. This are qtdeclarative-render2d-plugin and qtvirtualkeyboard.

declarative-render2d-plugin makes use of the Raster paint engine instead of OpenGL to render the  contents of a scene graph, thus making it useful when Qt Quick2 applications  are run in a system without OpenGL 2  enabled hardware. Using it might require tweaking Debian's /etc/X11/Xsession.d/90qt5-opengl. On Qt 5.9 and newer this plugin is merged in Qt GUI so there should be no need to perform any action on the user's behalf.

Debian's VirtualKeyboard currently has a gotcha: we are not building it with the embedded code it ships. Upstream ships 3rd party code but lacks a way to detect and use the system versions of them. See QTBUG-59594, patches are welcomed. Please note that we prefer patches sent directly upstream to the current dev revision, we will be happy to backport patches if necessary.
Yes, this means no hunspell, openwnn, pinyin, tcime nor lipi-toolkit/t9write support.

Categories: FLOSS Project Planets

GSoC preparation - Plasma Toolicons - A journey down the rabbit hole

Planet KDE - Sat, 2017-06-24 18:00

Before writing about my actual Summer of Code experiences, I wanted to briefly share what I worked on before the official coding start.

Plasma toolicons

Can you see these ridiculously small toolicons next to the Media frame plasmoid? I’m talking about the icons which allow you to Resize, Rotate and Remove the Plasmoid. Compared with other icons, these are clearly too small.

So, as preparation for GSoC, I wanted to know why this happens, and what would be required to make them bigger. Thus my journey into the (Plasma) rabbithole began…

Tracking down where the containment is defined was relatively easy, and shortly after i found the code responsible for the ActionButton:

PlasmaCore.ToolTipArea { id: button location: PlasmaCore.Types.LeftEdge mainText: action !== undefined ? action.text : "" mainItem: toolTipDelegate //API property QtObject svg property alias elementId: icon.elementId property QtObject action property bool backgroundVisible: false property int iconSize: 32 [...]

Huh, the iconSize is 32px? Well, that was easy to fix, surely this should be set to units.iconSizes.small and this problem is gone…

… or so I thought. No, this didn’t improve the situation, back to square one.

Is it overwritten by the look and feel theme? plasma-workspace/lookandfeel/contents/components/ActionButton.qml at least doesn’t - and it also happens with the style set to Oxygen.

While looking at this, I also noticed that units.iconSizes.small returned 16px on my system. This seemed odd, because the scale factor was set to 1.8x, so I would have expected bigger icons.

Where is this icon size calculated? Ah yes, in the file units.cpp, method Units::devicePixelIconSize.

int Units::devicePixelIconSize(const int size) const { /* in kiconloader.h enum StdSizes { SizeSmall=16, SizeSmallMedium=22, SizeMedium=32, [...] }; */ // Scale the icon sizes up using the devicePixelRatio // This function returns the next stepping icon size // and multiplies the global settings with the dpi ratio. const qreal ratio = devicePixelRatio(); if (ratio < 1.5) { return size; } else if (ratio < 2.0) { return size * 1.5; } else if (ratio < 2.5) { [...] }

Ok, my devicePixelRatio is 1.8 and therefore the icon size of a small pixmap gets multiplied by 1.5 and a request for a small (16px) pixmap should return a 24px pixmap.

But it doesn’t…

Debugging suggested that my devicePixelRatio is NOT 1.8, but rather around 1.4. How did that happen, isn’t the scale factor from the KDE settings used?

Oh, the comment in updateDevicePixelRatio() mentions that QGuiApplication::devicePixelRatio() is really not used:

void Units::updateDevicePixelRatio() { // Using QGuiApplication::devicePixelRatio() gives too coarse values, // i.e. it directly jumps from 1.0 to 2.0. We want tighter control on // sizing, so we compute the exact ratio and use that. // TODO: make it possible to adapt to the dpi for the current screen dpi // instead of assuming that all of them use the same dpi which applies for // X11 but not for other systems. QScreen *primary = QGuiApplication::primaryScreen(); if (!primary) { return; } const qreal dpi = primary->logicalDotsPerInchX(); // Usual "default" is 96 dpi // that magic ratio follows the definition of "device independent pixel" by Microsoft m_devicePixelRatio = (qreal)dpi / (qreal)96;

Hmm, yes, that was the case in earlier Qt versions when devicePixelRatio still returned an integer - but nowadays the value is a real number.

So, instead of calculating dpi / 96 I just changed it to return primary->devicePixelRatio().

Which now, finally, should return a devicePixelRatio of 1.8 and therefore result in bigger pixmaps.

Compiled it, and confident of victory restarted plasmashell… only to notice, that it still didn’t work.

What could there still go wrong?

So I got back to debugging.. to notice that primary->devicePixelRatio() returns a scale factor of 1.0. Huh? Isn’t this supposed to just use the QT_SCREEN_SCALE_FACTORS environment variable, which gets set to the value of the “Scale Display” dialog in the Systemsettings? If you want to know, the code for setting the environment variable is located in plasma-workspace/startkde/startplasmacompositor.cmake.

But why isn’t the problem gone, is there something in Plasma that overwrites this value?

Yes, of course there is!

The only way this value can be overwritten is due to the Qt attribute Qt::AA_DisableHighDpiScaling.

Grep’ing for that one pointed me to plasma-workspace/shell/main.cpp - the base file for plasmashell:

int main(int argc, char *argv[]) { // Devive pixel ratio has some problems in plasmashell currently. // - dialog continually expands (347951) // - Text element text is screwed (QTBUG-42606) // - Panel struts (350614) // This variable should possibly be removed when all are fixed qunsetenv("QT_DEVICE_PIXEL_RATIO"); QCoreApplication::setAttribute(Qt::AA_DisableHighDpiScaling);

I looked into the mentioned bugs. What should I do now? Re-enabling the HighDpiScaling-flag so the real devicePixelRatio is returned in Qt, and therefore I can use this value to calculate the sizes icons should be and then have the bigger Plasma ToolIcons? At least QTBUG-42606 seems to be fixed…

Oh boy, what have I gotten into now…

It was time to talk to my mentor!

David Edmundson quickly noticed that there should be no mismatch with the dpi / 96 calculation. Something fishy seems to be going on here…

What is this dpi value anyway? This is the one reported by xrdb -query |grep Xft.dpi and managed in the file $HOME/.config/kcmfonts.

And that value - as you probably can guess by now - did not make any sense. It didn’t match the expectation of being scaleFactor * 96, the value it should have been set to.

On we go to the location where the scaling configuration is set - scalingconfig.cpp in the KScreen KConfig Module.

void ScalingConfig::accept() { [...] fontConfigGroup.writeEntry("forceFontDPI", scaleDPI()); [...] } qreal ScalingConfig::scaleDPI() const { return scaleFactor() * 96.0; }

This scaleDPI is then applied with xrdb -quiet -merge -nocpp on startup.

So Xft.dpi is set to 1.8 * 96.0, or 172.8.

Have you spotted what is going wrong?

I did not, but David noticed…

The X server can only handle real values - and therefore 172.8 is simlpy discarded!

A few moments later this patch was ready…

… and I was finally able to enjoy my icons in all their scaled glory! And you can too, because this patch is already in Plasma 5.10.

Categories: FLOSS Project Planets

Steve Kemp: Linux security modules, round two.

Planet Debian - Sat, 2017-06-24 17:00

So recently I wrote a Linux Security Module (LSM) which would deny execution of commands, unless an extended attribute existed upon the filesystem belonging to the executables.

The whitelist-LSM worked well, but it soon became apparent that it was a little pointless. Most security changes are pointless unless you define what you're defending against - your "threat model".

In my case it was written largely as a learning experience, but also because I figured it seemed like it could be useful. However it wasn't actually as useful because you soon realize that you have to whitelist too much:

  • The redis-server binary must be executable, to the redis-user, otherwise it won't run.
  • /usr/bin/git must be executable to the git user.

In short there comes a point where user alice must run executable blah. If alice can run it, then so can mallory. At which point you realize the exercise is not so useful.

Taking a step back I realized that what I wanted to to prevent was the execution of unknown/unexpected, and malicious binaries How do you identify known-good binaries? Well hashes & checksums are good. So for my second attempt I figured I'd not look for a mere "flag" on a binary, instead look for a valid hash.

Now my second LSM is invoked for every binary that is executed by a user:

  • When a binary is executed the sha1 hash is calculated of the files contents.
  • If that matches the value stored in an extended attribute the execution is permitted.
    • If the extended-attribute is missing, or the checksum doesn't match, then the execution is denied.

In practice this is the same behaviour as the previous LSM - a binary is either executable, because there is a good hash, or it is not, because it is missing or bogus. If somebody deploys a binary rootkit this will definitely stop it from executing, but of course there is a huge hole - scripting-languages:

  • If /usr/bin/perl is whitelisted then /usr/bin/perl /tmp/exploit.pl will succeed.
  • If /usr/bin/python is whitelisted then the same applies.

Despite that the project was worthwhile, I can clearly describe what it is designed to achieve ("Deny the execution of unknown binaries", and "Deny binaries that have been modified"), and I learned how to hash a file from kernel-space - which was surprisingly simple.

(Yes I know about IMA and EVM - this was a simple project for learning purposes. Public-key signatures will be something I'll look at next/soon/later. :)

Perhaps the only other thing to explore is the complexity in allowing/denying actions based on the user - in a human-readable fashion, not via UIDs. So www-data can execute some programs, alice can run a different set of binaries, and git can only run /usr/bin/git.

Of course down that path lies apparmour, selinux, and madness..

Categories: FLOSS Project Planets

Stephen Ferg: Python Decorators

Planet Python - Sat, 2017-06-24 14:52

In August 2009, I wrote a post titled Introduction to Python Decorators. It was an attempt to explain Python decorators in a way that I (and I hoped, others) could grok.

Recently I had occasion to re-read that post. It wasn’t a pleasant experience — it was pretty clear to me that the attempt had failed.

That failure — and two other things — have prompted me to try again.

  • Matt Harrison has published an excellent e-book Guide to: Learning Python Decorators.
  • I now have a theory about why most explanations of decorators (mine included) fail, and some ideas about how better to structure an introduction to decorators.

There is an old saying to the effect that “Every stick has two ends, one by which it may be picked up, and one by which it may not.” I believe that most explanations of decorators fail because they pick up the stick by the wrong end.

In this post I will show you what the wrong end of the stick looks like, and point out why I think it is wrong. And I will show you what I think the right end of the stick looks like.


The wrong way to explain decorators

Most explanations of Python decorators start with an example of a function to be decorated, like this:

def aFunction(): print("inside aFunction")

and then add a decoration line, which starts with an @ sign:

@myDecorator def aFunction(): print("inside aFunction")

At this point, the author of the introduction often defines a decorator as the line of code that begins with the “@”. (In my older post, I called such lines “annotation” lines. I now prefer the term “decoration” line.)

For instance, in 2008 Bruce Eckel wrote on his Artima blog

A function decorator is applied to a function definition by placing it on the line before that function definition begins.

and in 2004, Phillip Eby wrote in an article in Dr. Dobb’s Journal

Decorators may appear before any function definition…. You can even stack multiple decorators on the same function definition, one per line.

Now there are two things wrong with this approach to explaining decorators. The first is that the explanation begins in the wrong place. It starts with an example of a function to be decorated and an decoration line, when it should begin with the decorator itself. The explanation should end, not start, with the decorated function and the decoration line. The decoration line is, after all, merely syntactic sugar — is not at all an essential element in the concept of a decorator.

The second is that the term “decorator” is used incorrectly (or ambiguously) to refer both to the decorator and to the decoration line. For example, in his Dr. Dobb’s Journal article, after using the term “decorator” to refer to the decoration line, Phillip Eby goes on to define a “decorator” as a callable object.

But before you can do that, you first need to have some decorators to stack. A decorator is a callable object (like a function) that accepts one argument—the function being decorated.

So… it would seem that a decorator is both a callable object (like a function) and a single line of code that can appear before the line of code that begins a function definition. This is sort of like saying that an “address” is both a building (or apartment) at a specific location and a set of lines (written in pencil or ink) on the front of a mailing envelope. The ambiguity may be almost invisible to someone familiar with decorators, but it is very confusing for a reader who is trying to learn about decorators from the ground up.


The right way to explain decorators

So how should we explain decorators?

Well, we start with the decorator, not the function to be decorated.

We start with the basic notion of a function — a function is something that generates a value based on the values of its arguments.

We note that in Python, functions are first-class objects, so they can be passed around like other values (strings, integers, objects, etc.).

We note that because functions are first-class objects in Python, we can write functions that both (a) accept function objects as argument values, and (b) return function objects as return values. For example, here is a function foobar that accepts a function object original_function as an argument and returns a function object new_function as a result.

def foobar(original_function): # make a new function def new_function(): # some code return new_function

We define “decorator”.

A decorator is a function (such as foobar in the above example) that takes a function object as an argument, and returns a function object as a return value.

So there we have it — the definition of a decorator. Anything else that we say about decorators is a refinement of, or an expansion of, or an addition to, this definition of a decorator.

We show what the internals of a decorator look like. Specifically, we show different ways that a decorator can use the original_function in the creation of the new_function. Here is a simple example.

def verbose(original_function): # make a new function that prints a message when original_function starts and finishes def new_function(*args, **kwargs): print("Entering", original_function.__name__) original_function(*args, **kwargs) print("Exiting ", original_function.__name__) return new_function

We show how to invoke a decorator — how we can pass into a decorator one function object (its input) and get back from it a different function object (its output). In the following example, we pass the widget_func function object to the verbose decorator, and we get back a new function object to which we assign the name talkative_widget_func.

def widget_func(): # some code talkative_widget_func = verbose(widget_func)

We point out that decorators are often used to add features to the original_function. Or more precisely, decorators are often used to create a new_function that does roughly what original_function does, but also does things in addition to what original_function does.

And we note that the output of a decorator is typically used to replace the original function that we passed in to the decorator as an argument. A typical use of decorators looks like this. (Note the change to line 4 from the previous example.)

def widget_func(): # some code widget_func = verbose(widget_func)

So for all practical purposes, in a typical use of a decorator we pass a function (widget_func) through a decorator (verbose) and get back an enhanced (or souped-up, or “decorated”) version of the function.

We introduce Python’s “decoration syntax” that uses the “@” to create decoration lines. This feature is basically syntactic sugar that makes it possible to re-write our last example this way:

@verbose def widget_func(): # some code

The result of this example is exactly the same as the previous example — after it executes, we have a widget_func that has all of the functionality of the original widget_func, plus the functionality that was added by the verbose decorator.

Note that in this way of explaining decorators, the “@” and decoration syntax is one of the last things that we introduce, not one of the first.

And we absolutely do not refer to line 1 as a “decorator”. We might refer to line 1 as, say, a “decorator invocation line” or a “decoration line” or simply a “decoration”… whatever. But line 1 is not a “decorator”.

Line 1 is a line of code. A decorator is a function — a different animal altogether.


Once we’ve nailed down these basics, there are a few advanced features to be covered.

  • We explain that a decorator need not be a function (it can be any sort of callable, e.g. a class).
  • We explain how decorators can be nested within other decorators.
  • We explain how decorators decoration lines can be “stacked”. A better way to put it would be: we explain how decorators can be “chained”.
  • We explain how additional arguments can be passed to decorators, and how decorators can use them.

Ten — A decorators cookbook

The material that we’ve covered up to this point is what any basic introduction to Python decorators would cover. But a Python programmer needs something more in order to be productive with decorators. He (or she) needs a catalog of recipes, patterns, examples, and commentary that describes / shows / explains when and how decorators can be used to accomplish specific tasks. (Ideally, such a catalog would also include examples and warnings about decorator gotchas and anti-patterns.) Such a catalog might be called “Python Decorator Cookbook” or perhaps “Python Decorator Patterns”.

So that’s it. I’ve described what I think is wrong (well, let’s say suboptimal) about most introductions to decorators. And I’ve sketched out what I think is a better way to structure an introduction to decorators.

Now I can explain why I like Matt Harrison’s e-book Guide to: Learning Python Decorators. Matt’s introduction is structured in the way that I think an introduction to decorators should be structured. It picks up the stick by the proper end.

The first two-thirds of the Guide hardly talk about decorators at all. Instead, Matt begins with a thorough discussion of how Python functions work. By the time the discussion gets to decorators, we have been given a strong understanding of the internal mechanics of functions. And since most decorators are functions (remember our definition of decorator), at that point it is relatively easy for Matt to explain the internal mechanics of decorators.

Which is just as it should be.

Revised 2012-11-26 — replaced the word “annotation” with “decoration”, following terminology ideas discussed in the comments.
Categories: FLOSS Project Planets

Ingo Juergensmann: Upgrade to Debian Stretch - GlusterFS fails to mount

Planet Debian - Sat, 2017-06-24 14:45

Before I upgrade from Jessie to Stretch everything worked as a charme with glusterfs in Debian. But after I upgraded the first VM to Debian Stretch I discovered that glusterfs-client was unable to mount the storage on Jessie servers. I got this in glusterfs log:

[2017-06-24 12:51:53.240389] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.8 (args: /usr/sbin/glusterfs --read-only --fuse-mountopts=nodev,noexec --volfile-server= --volfile-id=/le --fuse-mountopts=nodev,noexec /etc/letsencrypt.sh/certs)
[2017-06-24 12:51:54.534826] E [mount.c:318:fuse_mount_sys] 0-glusterfs-fuse: ret = -1

[2017-06-24 12:51:54.534896] I [mount.c:365:gf_fuse_mount] 0-glusterfs-fuse: direct mount failed (Invalid argument) errno 22, retry to mount via fusermount
[2017-06-24 12:51:56.668254] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-06-24 12:51:56.671649] E [glusterfsd-mgmt.c:1590:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server
[2017-06-24 12:51:56.671669] E [glusterfsd-mgmt.c:1690:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:/le)
[2017-06-24 12:51:57.014502] W [glusterfsd.c:1327:cleanup_and_exit] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_handle_reply+0x90) [0x7fbea36c4a20] -->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x494) [0x55fbbaed06f4] -->/usr/sbin/glusterfs(cleanup_and_exit+0x54) [0x55fbbaeca444] ) 0-: received signum (0), shutting down
[2017-06-24 12:51:57.014564] I [fuse-bridge.c:5794:fini] 0-fuse: Unmounting '/etc/letsencrypt.sh/certs'.
[2017-06-24 16:44:45.501056] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.8 (args: /usr/sbin/glusterfs --read-only --fuse-mountopts=nodev,noexec --volfile-server= --volfile-id=/le --fuse-mountopts=nodev,noexec /etc/letsencrypt.sh/certs)
[2017-06-24 16:44:45.504038] E [mount.c:318:fuse_mount_sys] 0-glusterfs-fuse: ret = -1

[2017-06-24 16:44:45.504084] I [mount.c:365:gf_fuse_mount] 0-glusterfs-fuse: direct mount failed (Invalid argument) errno 22, retry to mount via fusermount

After some searches on the Internet I found Debian #858495, but no solution for my problem. Some search results recommended to set "option rpc-auth-allow-insecure on", but this didn't help. In the end I joined #gluster on Freenode and got some hints there:

JoeJulian | ij__: debian breaks apart ipv4 and ipv6. You'll need to remove the ipv6 ::1 address from localhost in /etc/hosts or recombine your ip stack (it's a sysctl thing)
JoeJulian | It has to do with the decisions made by the debian distro designers. All debian versions should have that problem. (yes, server side).

Removing ::1 from /etc/hosts and from lo interface did the trick and I could mount glusterfs storage from Jessie servers in my Stretch VMs again. However, when I upgraded the glusterfs storages to Stretch as well, this "workaround" didn't work anymore. Some more searching on the Internet made me found this posting on glusterfs mailing list:

We had seen a similar issue and Rajesh has provided a detailed explanation on why at [1]. I'd suggest you to not to change glusterd.vol but execute "gluster volume set <volname> transport.address-family inet" to allow Gluster to listen on IPv4 by default.

Setting this option instantly fixed my issues with mounting glusterfs storages.

So, whatever is wrong with glusterfs in Debian, it seems to have something to do with IPv4 and IPv6. When disabling IPv6 in glusterfs, it works. I added information to #858495.

Kategorie: DebianTags: DebianGlusterFSSoftwareServer 
Categories: FLOSS Project Planets

Janusworx: On Starting Summer Training at #dgplug

Planet Python - Sat, 2017-06-24 14:33

I started out with a very vague idea, of learning programming last year.

I went to Pycon India, fell in love with the community, decided to learn software, and came home all charged up.
(Btw, I was so intimidated, I did not speak to a single soul)

The plan was to sort personal issues, tackle a couple of major work projects so that I could then focus on learning, clear the decks and go full steam ahead come April.

While I made headway, I was also missing the hum and bustle of Pycon that had so charged me, but I did remember one session I attended, that had left me smiling was a sponsored talk of all things, by a certain Mr. Das. Off the cuff, naturally, warmly delivered.

So as I was looking for … someone to talk to, somewhere to belong, who comes along but Santa Das.

While that trip didn't quite happen due to personal reasons, we still kept in touch.
(Why he would do that with a newbie-know-nothing like me, I don’t know. The man has a large heart)

And when the new session of #dgplug was announced, I jumped at the chance!

To those not part of the dgplug summer training, read all about it here.
The brave1 souls at the Linux Users’ Group of Durgapur take in a bunch of kids (and adults) who want to learn all about the magical world of software programming and give them tools with which they can paint on that vast canvas.

Our goal is to bring in more upstream contributors to various FOSS projects.
Through this training we show the path of becoming an upstream contributor.

— from the DGPLUG summer training page

Communication skills, free software projects, documentation, system administration, source code management, time management, conference proposals and obviously basic programming – the whole gamut is covered here

So while any odd duck can learn on their own, the DGPLUG summer sessions will help you become a well rounded individual who can code and contribute to the world.
A software finishing school, if you will :)

Kushal and the training and it’s successes have been featured in opensource.com time and time again

A look at the guest speakers (including the all father of Python and the cream of the Indian Developer community) should be enough to convince you to come join.

It’s only been a week, and I’ve been having a ball!
We covered communication skills, touch typing and the vi editor this week!
If you hurry, you can catch up and work with us.

And for my new #dgplug family, here’s a little something, something2 about me to close

  1. Yes, I am obviously hiding my big, fat tummy in the pic :) 3
  2. I’m like a poor man’s, still failing James Altucher.
  3. Yes, I’m a lot older than most of you :) 4
  4. I’ve been at this IT thing a long time. (since 1997, in fact) 5
  5. And yes, only now do I get the bright idea to learn software.
  6. I love the fact, that I get you to be my plus-minus-equal
  7. You folks make me feel all warm and enthusiastic and welcoming and make me feel like I found my tribe!
  8. I’m still head over heels in love with my better half

I look to learn so much from you and know so much more of you over the coming months.
I wish you all make good art!

  1. (& foolhardy, dare I say :P )

  2. My grandma says that :)

  3. dropped 7 kgs to 89. Only another 20 to go!

  4. not necessarily wiser :P

  5. land line telephone fixer man, hardware tech support at small firm, hardware tech support at huge firm, freelance engineer, consulting engineer, consulting manager.

Categories: FLOSS Project Planets

Latte Dock accepts donations, what is coming...

Planet KDE - Sat, 2017-06-24 14:12

Hello again...
because there are users that requested a way in order to support Latte even further and donate for the project, we created a donation page at pledgie: https://pledgie.com/campaigns/34116

and we added also a donate button in our main page at: https://github.com/psifidotos/Latte-Dock/

to cheer you up a bit for the upcoming 0.7 version which is scheduled for the end of August or maybe earlier ;) based on the effort...
some of the features already implemented and some that are going to land:
  • support layouts and change them with one click (we enhanced our export/import configuration mechanism to layouts. We provide already 5 predefined layouts and the user can add as many wants to)
  • provide two layouts for the configuration window (Basic/Advanced)
  • set the dock transparency per dock and enable/disable its panel shadows
  • global shortcuts, Super+No (unity way), Super+Ctrl+No (new instance), Super+` (raise the hidden dock)
  • support from libtaskmanager the libunity interface in order to show progress indicators and item counters such as unread e-mails etc... this needs libunity9 installed. +expose independently our own simpler dbus interface to show counters for tasks if there are programmers out there that dont want to use the libunity way
  • tasks audio streams indicator (increase/decrease volume with scroll whell and mute with clicking it)
  • the new Places exposer for your tasks/launchers that plasma 5.10 offers
  • dynamic dock background (show the background only for maximized windows and choose also if you want the dock shadow to be shown in that case)
  • copy your docks easily with one-click (extremely useful in multi-screen environments that want the same dock between different screens)
  • sync your launchers between all your docks (global launchers, they do not erase your per dock launchers if you disable them)
  • wayland tech preview (there is already code implemented for this and landed in our master)
  • support fillWidth/Height applets like the plasma panel
  • substitute the Latte plasmoid with your favourite plasma taskmanager (having implemented proper fillWidth/Height for the applets we can now provide this)
  • support separators everywhere and work correclty with the parabolic effect
  • various improvements for launchers start up, animations etc....

Latte 0.7 will be compatible with plasma>=5.9 and qt>=5.7

thanks everyone once more for your support...

Categories: FLOSS Project Planets
Syndicate content