FLOSS Project Planets

LibreOffice the better Office, really?

Planet KDE - Wed, 2017-04-26 07:53
Idea and concept

See the previous blog post LibreOffice the better Office.

Have a look

Test it!

Download

  • dev-builds.libreoffice.org/daily/master/

Configure

  • LO Writer -> Tools -> Options -> Advanced: Enable experimental features
  • View -> Toolbar Layout -> enable Notebookbar
  • View -> Notebookbar -> enable Groupedbar Full

Open Bugs

  • Icon size: The layout follow the settings in Tools -> Options -> View -> Notebookbar icon size style in general but a view icons are (for now) fixed size so change the Notebookbar icon size to Large TDF#107259
  • Background color: it is possible to theme the toolbar with a firefox theme Tools -> Options -> Personalization -> Select Theme -> https://addons.mozilla.org/de/firefox/addon/libreoffice-groupbar-full/ In the future background theming should be possible to. TDF#107128
  • Different sizes: The toolbar is defined for 1400 to 1700 px in width in the future it will be possible to make the toolbar available for width. We love google summer of code
Feedback

There are way more issues, bugs, missing features, please comment, make a bug report or report to the Contextual Group Toolbar bug TDF#106035


Categories: FLOSS Project Planets

A. Jesse Jiryu Davis: Grok the GIL: Write Fast and Thread-Safe Python

Planet Python - Wed, 2017-04-26 07:52

(Cross-posted from Red Hat’s OpenSource.com.)

When I was six years old I had a music box. I’d wind it up, and a ballerina revolved on top of the box while a mechanism inside plinked out “Twinkle Twinkle Little Star.” The thing must have been godawful tacky, but I loved it, and I wanted to know how it worked. Somehow I got it open and was rewarded with the sight of a simple device, a metal cylinder the size of my thumb, studded so that as it rotated, it plucked the teeth of a steel comb and made the notes.

Of all a programmer’s traits, curiosity about how things work is the sine qua non. When I opened my music box to see inside, I showed that I could grow up to be, if not a great programmer, then at least a curious one.

It is odd, then, that for many years I wrote Python programs while holding mistaken notions about the Global Interpreter Lock, because I was never curious enough to look at how it worked. I’ve met others with the same hesitation, and the same ignorance. The time has come for us to pry open the box. Let’s read the CPython interpreter source code and find out exactly what the GIL is, why Python has one, and how it affects your multithreaded programs. I’ll show some examples to help you grok the GIL. You will learn to write fast and thread-safe Python, and how to choose between threads and processes.

(And listen, my dear pedantic reader: for the sake of focus, I only describe CPython here, not Jython, PyPy, or IronPython. If you would like to learn more about those interpreters I encourage you to research them. I restrict this article to the Python implementation that working programmers overwhelmingly use.)

Behold, the Global Interpreter Lock

Here it is:

static PyThread_type_lock interpreter_lock = 0; /* This is the GIL */

This line of code is in ceval.c, in the CPython 2.7 interpreter’s source code. Guido van Rossum’s comment, “This is the GIL,” was added in 2003, but the lock itself dates from his first multithreaded Python interpreter in 1994. On Unix systems, PyThread_type_lock is an alias for the standard C lock, mutex_t. It is initialized when the Python interpreter begins:

void PyEval_InitThreads(void) { interpreter_lock = PyThread_allocate_lock(); PyThread_acquire_lock(interpreter_lock); }

All C code within the interpreter must hold this lock while executing Python. Guido first built Python this way because it is simple, and every attempt to remove the GIL from CPython has cost single-threaded programs too much performance to be worth the gains for multithreading.

The GIL’s effect on the threads in your program is simple enough that you can write the principle on the back of your hand: “One thread runs Python, while N others sleep or await I/O.” Python threads can also wait for a threading.Lock or other synchronization object from the threading module; we shall consider threads in that state to be “sleeping”, too.

When do threads switch? Whenever a thread begins sleeping or awaiting network I/O, there is a chance for another thread to take the GIL and execute some Python code. This is “cooperative multitasking.” CPython also has “preemptive multitasking”: If a thread runs uninterrupted for 1000 bytecode instructions in Python 2, or runs 15 milliseconds in Python 3, then it gives up the GIL and another thread may run. Think of this like “time slicing” in the olden days when we had many threads but one CPU. We shall discuss these two kinds of multitasking in detail.

Think of Python as an old mainframe: many tasks share one CPU.

Cooperative multitasking

When it begins a task, such as network I/O, that is of long or uncertain duration and does not require running any Python code, a thread relinquishes the GIL so another thread can take it and run Python. This polite conduct is called cooperative multitasking, and it allows concurrency: many threads can wait for different events at the same time.

Say that two threads each connect a socket:

def do_connect(): s = socket.socket() s.connect(('python.org', 80)) # drop the GIL for i in range(2): t = threading.Thread(target=do_connect) t.start()

Only one of these two threads can execute Python at a time, but once it has begun connecting, it drops the GIL so the other thread can run. This means that both threads could be waiting for their sockets to connect concurrently. This is a good thing! They can do more work in the same amount of time.

Let us pry open the box and see how a Python thread actually drops the GIL while it waits for a connection to be established, in socketmodule.c:

/* s.connect((host, port)) method */ static PyObject * sock_connect(PySocketSockObject *s, PyObject *addro) { sock_addr_t addrbuf; int addrlen; int res; /* convert (host, port) tuple to C address */ getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen); Py_BEGIN_ALLOW_THREADS res = connect(s->sock_fd, addr, addrlen); Py_END_ALLOW_THREADS /* error handling and so on .... */ }

The Py_BEGIN_ALLOW_THREADS macro is where the thread drops the GIL, it is simply defined as:

PyThread_release_lock(interpreter_lock);

And of course Py_END_ALLOW_THREADS reacquires the lock. A thread might block at this spot, waiting for another thread to release the lock; once that happens, the waiting thread grabs the GIL back and resumes executing your Python code. In short: While N threads are blocked on network I/O or waiting to reacquire the GIL, one thread can run Python.

Below, we’ll see a complete example that uses cooperative multitasking to quickly fetch many URLs. But before that, let us contrast cooperative multitasking with the other kind of multitasking.

Preemptive multitasking

A Python thread can voluntarily release the GIL, but it can also have the GIL seized from it preemptively.

Let’s back up and talk about how Python is executed. Your program is run in two stages. First, your Python text is compiled into a simpler binary format called bytecode. Second, the Python interpreter’s main loop, a function mellifluously named PyEval_EvalFrameEx, reads the bytecode and executes the instructions in it one by one.

While the interpreter steps through your bytecode it periodically drops the GIL, without asking permission of the thread whose code it is executing, so other threads can run:

for (;;) { if (--ticker < 0) { ticker = check_interval; /* Give another thread a chance */ PyThread_release_lock(interpreter_lock); /* Other threads may run now */ PyThread_acquire_lock(interpreter_lock, 1); } bytecode = *next_instr++; switch (bytecode) { /* execute the next instruction ... */ } }

By default the check interval is 1000 bytecodes. All threads run this same code and have the lock taken from them periodically in the same way. In Python 3 the GIL’s implementation is more complex, and the check interval is not a fixed number of bytecodes but 15 milliseconds. For your code, however, these differences are not significant.

Thread safety in Python

Weaving together multiple threads requires skill.

If a thread can lose the GIL at any moment, you must make your code thread-safe. Python programmers think differently about thread safety than C or Java programmers do, however, because many Python operations are “atomic”.

An example of an atomic operation is calling sort() on a list. A thread cannot be interrupted in the middle of sorting, and other threads never see a partly-sorted list, nor see stale data from before the list was sorted. Atomic operations simplify our lives, but there are surprises. For example, += seems simpler than sort(), but += is not atomic! How can you know which operations are atomic and which are not?

Consider this code:

n = 0 def foo(): global n n += 1

We can see the bytecode to which this function compiles, with Python’s standard dis module:

>>> import dis >>> dis.dis(foo) LOAD_GLOBAL 0 (n) LOAD_CONST 1 (1) INPLACE_ADD STORE_GLOBAL 0 (n)

One line of code, n += 1, has been compiled to four bytecodes, which do four primitive operations:

  1. load the value of n onto the stack
  2. load the constant 1 onto the stack
  3. sum the two values at the top of the stack
  4. store the sum back into n

Remember that, every 1000 bytecodes, a thread is interrupted by the interpreter taking the GIL away. If the thread is unlucky this might happen between the time it loads the value of n onto the stack and when it stores it back. It is easy see how this leads to lost updates:

threads = [] for i in range(100): t = threading.Thread(target=foo) threads.append(t) for t in threads: t.start() for t in threads: t.join() print(n)

Usually this code prints “100”, because each of the 100 threads has incremented n. But sometimes you see 99 or 98, if one of the threads’ updates was overwritten by another.

So, despite the GIL, you still need locks to protect shared mutable state:

n = 0 lock = threading.Lock() def foo(): global n with lock: n += 1

What if we were using an atomic operation like sort instead?:

lst = [4, 1, 3, 2] def foo(): lst.sort()

This function’s bytecode shows that sort cannot be interrupted, because it is atomic:

>>> dis.dis(foo) LOAD_GLOBAL 0 (lst) LOAD_ATTR 1 (sort) CALL_FUNCTION 0

The one line compiles to three bytecodes:

  1. load the value of lst onto the stack
  2. load its sort method onto the stack
  3. call the sort method

Even though the line lst.sort() takes several steps, the sort call itself is a single bytecode. If we inspect the CPython source for sort we see it’s a C function that doesn’t call any Python code or drop the GIL, so long as we haven’t provided a Python callback for the key parameter. Thus, there is no opportunity for the thread to have the GIL seized from it during the call. We could conclude that we don’t need to lock around sort. Or, to avoid worrying about which operations are atomic, we can follow a simple rule: always lock around reads and writes of shared mutable state. After all, acquiring a threading.Lock in Python is cheap.

(Following this simple rule will also improve your code’s chances of running correctly in interpreters without a GIL, like Jython or IronPython, in which operations like list.sort are not atomic.)

Although the GIL does not excuse us from the need for locks, it does mean there is no need for fine-grained locking in CPython. In a free-threaded language like Java, programmers make an effort to lock shared data for the shortest time possible, to reduce thread contention and allow maximum parallelism. Since threads cannot run Python in parallel, however, there’s no advantage to fine-grained locking. So long as no thread holds a lock while it sleeps, does I/O, or some other GIL-dropping operation, you should use the coarsest, simplest locks possible. Other threads couldn’t have run in parallel anyway.

Use simple locks in Python.

Finishing Sooner With Concurrency

I wager what you really came for is to optimize your programs with multi-threading. If your task will finish sooner by awaiting many network operations at once, then multiple threads help, even though only one of them can execute Python at a time. This is “concurrency”, and threads work nicely in this scenario.

This code runs faster with threads:

import threading import requests urls = [...] def worker(): while True: try: url = urls.pop() except IndexError: break # Done. requests.get(url) for _ in range(10): t = threading.Thread(target=worker) t.start()

As we saw above, these threads drop the GIL while waiting for each socket operation involved in fetching a URL over HTTP, so they finish the work sooner than a single thread could.

Parallelism

What if your task will finish sooner only by running Python code simultaneously? This kind of scaling is called “parallelism”, and the GIL prohibits it. You must use multiple processes. It can be more complicated than threading and requires more memory, but it will take advantage of multiple CPUs.

This example finishes sooner by forking 10 processes than it could with only one, because the processes run in parallel on several cores:

import os import sys nums = [1 for _ in range(1000000)] chunk_size = len(nums) // 10 readers = [] while nums: chunk, nums = nums[:chunk_size], nums[chunk_size:] reader, writer = os.pipe() if os.fork(): readers.append(reader) # Parent. else: subtotal = 0 for i in chunk: # Intentionally slow code. subtotal += i print('subtotal %d' % subtotal) os.write(writer, str(subtotal).encode()) sys.exit(0) # Parent. total = 0 for reader in readers: subtotal = int(os.read(reader, 1000).decode()) total += subtotal print("Total: %d" % total)

This wouldn’t run faster with 10 threads than with one, because only one thread can execute Python at a time. But since it forks 10 processes, and each forked process has a separate GIL, this program can parcel the work out and run multiple computations at once.

(Jython and IronPython provide single-process parallelism but they are very far from full CPython compatibility. PyPy with Software Transactional Memory may some day be fast. Try these interpreters if you’re curious.)

Conclusion

Now that you’ve opened the music box and seen the simple mechanism, you know all you need to write fast, thread-safe Python. Use threads for concurrent I/O, and processes for parallel computation. The principle is plain enough that you might not even need to write it on your hand!

Categories: FLOSS Project Planets

PythonClub - A Brazilian collaborative blog about Python: What the Flask? pt 4 - Extensões para o Flask

Planet Python - Wed, 2017-04-26 06:41
What The Flask - 4/5

Finalmente!!! Depois de uma longa espera o What The Flask está de volta! A idéia era publicar primeiro a parte 4 (sobre Blueprints) e só depois a 5 sobre como criar extensões. Mas esses 2 temas estão muito interligados então neste artigo os 2 assuntos serão abordados. E a parte 5 será a final falando sobre deploy!

  1. Hello Flask: Introdução ao desenvolvimento web com Flask
  2. Flask patterns: Estruturando aplicações Flask
  3. Plug & Use: extensões essenciais para iniciar seu projeto
  4. Magic(app): Criando Extensões para o Flask(<-- Você está aqui)
  5. Run Flask Run: "deploiando" seu app nos principais web servers e na nuvem

Não sei se você ainda se lembra? mas estavámos desenvolvendo um CMS de notícias, utilizamos as extensões Flask-MongoEngine, Flask-Security, Flask-Admin e Flask-Bootstrap.

E neste artigo iremos adicionar mais uma extensão em nosso CMS, mas iremos criar uma extensão ao invés de usar uma das extensões disponíveis.

Extensão ou Plugin? Por definição plugins diferem de extensões. Plugins geralmente são externos e utilizam algum tipo de API pública para se integrar com o aplicativo. Extensões, por outro lado, geralmente são integradas com a lógica da aplicação, isto é, as interfaces do próprio framework. Ambos, plugins e extensões, aumentam a utilidade da aplicação original, mas plugin é algo relacionado apenas a camadas de mais alto nível da aplicação, enquanto extensões estão acopladas ao framework. Em outras palavras, plugin é algo que você escreve pensando apenas na sua aplicação e está altamente acoplado a ela enquanto extensão é algo que pode ser usado por qualquer aplicação escrita no mesmo framework pois está acoplado a lógica do framework e não das aplicações escritas com ele.

Quando criar uma extensão?

Faz sentido criar uma extensão quando você identifica uma functionalidade que pode ser reaproveitada por outras aplicações Flask, assim você mesmo se beneficia do fato de não precisar reescrever (copy-paste) aquela funcionalidade em outros apps e também pode publicar sua extensão como open-source beneficiando toda a comunidade e incorporando as melhorias, ou seja, todo mundo ganha!

Exemplo prático

Imagine que você está publicando seu site mas gostaria de prover um sitemap. (URL que lista todas as páginas existentes no seu site usada pelo Google para melhorar a sua classificação nas buscas).

Como veremos no exemplo abaixo publicar um sitemap é uma tarefa bastante simples, mas é uma coisa que você precisará fazer em todos os sites que desenvolver e que pode se tornar uma funcionalidade mais complexa na medida que necessitar controlar datas de publicação e extração de URLs automáticamente.

Exemplo 1 - Publicando o sitemap sem o uso de extensões from flask import Flask, make_response app = Flask(__name__) @app.route('/artigos') def artigos(): "este endpoint retorna a lista de artigos" @app.route('/paginas') def paginas(): "este endpoint retorna a lista de paginas" @app.route('/contato') def contato(): "este endpoint retorna o form de contato" ###################################### # Esta parte poderia ser uma extensão ###################################### @app.route('/sitemap.xml') def sitemap(): items = [ '<url><loc>{0}</loc></url>'.format(page) for page in ['/artigos', '/paginas', '/contato'] ] sitemap_xml = ( '<?xml version="1.0" encoding="UTF-8"?>' '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">{0}</urlset>' ).format(''.join(items)).strip() response = make_response(sitemap_xml) response.headers['Content-Type'] = 'application/xml' return response ####################################### # / Esta parte poderia ser uma extensão ####################################### app.run(debug=True)

Executando e acessando http://localhost:5000/sitemap.xml o resultado será:

<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url><loc>/artigos</loc></url> <url><loc>/paginas</loc></url> <url><loc>/contato</loc></url> </urlset>

NOTE: O app acima é apenas um simples exemplo de como gerar um sitemap e a intenção dele aqui é apenas a de servir de exemplo para extensão que criaremos nos próximos passos, existem outras boas práticas a serem seguidas na publicação de sitemap mas não é o foco deste tutorial.

Vamos então transformar o exemplo acima em uma Extensão e utilizar uma abordagem mais dinâmica para coletar as URLs, mas antes vamos entender como funcionam as extensões no Flask.

Como funciona uma Extensão do Flask?

Lembra que na parte 2 desta série falamos sobre os patterns do Flask e sobre o application factory e blueprints? As extensões irão seguir estes mesmos padrões em sua arquitetura.

O grande "segredo" para se trabalhar com Flask é entender que sempre iremos interagir com uma instância geralmente chamada de app e que pode ser acessada também através do proxy current_app e que sempre aplicaremos um padrão que é quase funcional neste deste objeto sendo que a grande diferença aqui é que neste paradigma do Flask as funções (chamadas de factories) introduzem side effects, ou seja, elas alteram ou injetam funcionalidades no app que é manipulado até que chega ao seu estado de execução. (enquanto em um paradigma funcional as funções não podem ter side effects)

Também é importante entender os estados configuração, request e interativo/execução do Flask, asunto que abordamos na parte 1 desta série.

Em resumo, iremos criar uma factory que recebe uma instância da classe Flask, o objeto app (ou o acessa através do proxy current_app) e então altera ou injeta funcionalidades neste objeto.

Dominar o Flask depende muito do entendimento desse padrão de factories e os 3 estados da app citados acima, se você não está seguro quanto a estes conceitos aconselho reler as partes 1 e 2 desta série (e é claro sinta se livre para deixar comentários com as suas dúvidas).

Só para relembrar veja o seguinte exemplo:

app = Flask(__name__) # criamos a instancia de app admin = Admin() # instancia do Flask-Admin ainda não inicializada do_something(app) # injeta ou altera funcionalidades do app do_another_thing(app, admin) # injeta ou altera funcionalidades do apo ou do admin Security(app) # adiciona funcionalidades de login e controle de acesso Cache(app) # adiciona cache para views e templates SimpleSitemap(app) # A extensão que iremos criar! Ela adiciona o /sitemap.xml no app admin.init_app(app) # agora sim inicializamos o flask-admin no modo lazy

Olhando o código acima pode parecer bastante simples, você pode achar que basta receber a intância de app e sair alterando sem seguir nenhum padrão.

def bad_example_of_flask_extension(app): "Mal exemplo de factory que injeta ou altera funcionalidades do app" # adiciona rotas @app.route('/qualquercoisa) def qualquercoisa(): ... # substitui objetos usando composição app.config_class = MyCustomConfigclass # altera config app.config['QUALQUERCOISA'] = 'QUALQUERVALOR' # sobrescreve métodos e atributos do app app.make_responde = another_function # Qualquer coisa que o Python (e a sua consciência) permita!

Isso pode provavelmente funcionar mas não é uma boa prática, existem muitos problemas com o factory acima e alguns deles são:

  1. Nunca devemos definir rotas dessa maneira com app.route em uma extensão o correto é usar blueprints.
  2. Lembre-se dos 3 estados do Flask, devemos levar em consideração que no momento que a aplicação for iniciada a extensão pode ainda não estar pronta para ser carregada, por exemplo, a sua extensão pode depender de um banco de dados que ainda não foi inicializado, portanto as extensões precisam sempre ter um modo lazy.
  3. Usar funções pode se ruma boa idéia na maioria dos casos, mas lembre-se que precisamos manter estado em algumas situações então pode ser melhor usar classes ao invés de funções pois as classes permitem uma construção mais dinâmica.
  4. Nossa extensão precisa ser reutilizavel em qualquer app flask, portanto devemos usar namespaces ao ler configurações e definir rotas.

NOTE: Eu NÃO estou dizendo que você não deve usar funções para extender seu app Flask, eu mesmo faço isso em muitos casos. Apenas tenha em mente esses detalhes citados na hora de decidir qual abordagem usar.

Patterns of a Flask extension

Preferencialmente uma Extensão do Flask deve seguir esses padrões:

  • Estar em um módulo nomeado com o prefixo flask_ como por exemplo flask_admin e flask_login e neste artigo criaremos o flask_simple_sitemap. (NOTE: Antigamente as extensões usavam o padrão flask.ext.nome_extensao mas este tipo de plublicação de módulo com namespace do flask.ext foi descontinuado e não é mais recomendado.)
  • Fornecer um método de inicialização lazy nomeado init_app.
  • Ler todas as suas configurações a partir do app.config
  • Ter suas configurações prefixadas com o nome da extensão, exemplo: SIMPLE_SITEMAP_URLS ao invés de apenas SITEMAP_URLS pois isto evita conflitos com configurações de outras extensões.
  • Caso a extensão adicione views e URL rules, isto deve ser feito com Blueprint
  • Caso a extensão adicione arquivos estáticos ou de template isto também deve ser feito com Blueprint
  • ao registrar urls e endpoints permitir que sejam dinâmicos através de config e sempre prefixar com o nome da extensão. Exemplo: url_for('simple_sitemap.sitemap') é melhor do que url_for('sitemap') para evitar conflitos com outras extensões.

NOTE: Tenha em mente que regras foram feitas para serem quebradas, O Guido escreveu na PEP8 "A Foolish Consistency is the Hobgoblin of Little Minds", ou seja, tenha os padrões como guia mas nunca deixe que eles atrapalhem o seu objetivo. Eu mesmo já quebrei essa regra 1 no flasgger, eu poderia ter chamado de flask_openapi ou flask_swaggerui mas achei Flasgger um nome mais divertido :)

De todos os padrões acima o mais importante é o de evitar o conflito com outras extensões!

Zen do Python: Namespaces são uma ótima idéia! vamos usar mais deles!

Criando a extensão Simple Sitemap

Ok, agora que você já sabe a teoria vamos colocar em prática, abre ai o vim, emacs, pycharm ou seu vscode e vamos reescrever o nosso app do Exemplo 1 usando uma extensão chamada flask_simple_sitemap e para isso vamos começar criando a extensão:

A extensão será um novo módulo Python que vai ser instalado usando setup ou pip portanto crie um projeto separado.

Em seu terminal *nix execute os comandos:

➤ $ # Entre na pasta onde vc armazena seus projetos cd Projects # crie o diretório root do projeto mkdir simple_sitemap cd simple_sitemap # torna a extensão instalável (vamos escrever esse arquivo depois) touch setup.py # é sempre bom incluir uma documentação básica echo '# Prometo documentar essa extensão!' > README.md # se não tiver testes não serve para nada! :) touch tests.py # crie o diretório que será o nosso package mkdir flask_simple_sitemap # __init__.py para transformar o diretório em Python package echo 'from .base import SimpleSitemap' > flask_simple_sitemap/__init__.py # A implementação da extensão será escrita neste arquivo # (evite usar main.py pois este nome é reservado para .zipped packages) touch flask_simple_sitemap/base.py # Crie a pasta de templates mkdir flask_simple_sitemap/templates # usaremos Jinja para gerar o XML touch flask_simple_sitemap/templates/sitemap.xml # incluindo templates no build manifest do setuptools echo 'recursive-include flask_simple_sitemap/templates *' > MANIFEST.in # precisaremos de um arquivo de requirements para os testes touch requirements-test.txt

Agora voce terá a seguinte estrutura:

➤ tree simple_sitemap/ ├── flask_simple_sitemap/ │   ├── base.py │   ├── __init__.py │   └── templates/ │   └── sitemap.xml ├── MANIFEST.in ├── README.md ├── requirements-test.txt ├── setup.py └── tests.py 2 directories, 8 files

O primeiro passo é escrever o setup.py já que a extensão precisa ser instalavél:

from setuptools import setup, find_packages setup( name='flask_simple_sitemap', version='0.0.1', packages=find_packages(), include_package_data=True, zip_safe=False )

Eu tenho o costumo de praticar o que eu chamo de RDD (Readme Driven Ddevelopment), ou seja, ao criar projetos como este eu costumo escreve primeiro o README.md explicando como deve funcionar e só depois de ter isto pronto que eu começo a programar.

Edite o README.md

# Flask Simple Sitemap Esta extensão adiciona a funcionalidade de geração de sitemap ao seu app flask. ## Como instalar? Para instalar basta clonar o repositório e executar: $ python setup.py install Ou via `pip install flask_simple_sitemap` ## Como usar? Basta importar e inicializar: from flask import Flask from flask_simple_sitemap import SimpleSitemap app = Flask(__name__) SimpleSitemap(app) @app.route('/) def index(): return 'Hello World' Como em toda extensão Flask também é possível inicializar no modo Lazy chamando o método `init_app` ## Opções de configuração: esta extensão utiliza o namespace de configuração `SIMPLE_SITEMAP_` - **SIMPLE_SITEMAP_BLUEPRINT** define o nome do blueprint e do url prefix (default: `'simple_sitemap'`) - **SIMPLE_SITEMAP_URL** define a url que irá renderizar o sitemap (default: `'/sitemap.xml'`) - **SIMPLE_SITEMAP_PATHS** dicionário de URLs a serem adicionadas ao sitemap (exemplo: URLs criadas a partir de posts em bancos de dados)

Agora que já sabemos pelo README o que queremos entregar de funcionalidade já é possível escrever o tests.py e aplicar também um pouco de TDD

O Flask tem uma integração bem interesante com o py.test e podemos editar o tests.py da seguinte maneira:

NOTE: O ideal é fazer o test setup no arquivo conftest.py e usar fixtures do py.test, mas aqui vamos escrever tudo junto no tests.py para ficar mais prático.

Zen do Python: praticidade vence a pureza :)

#################################################################### # Início do Test Setup # import xmltodict from flask import Flask from flask_simple_sitemap import SimpleSitemap app = Flask(__name__) extension = SimpleSitemap() app.config['SIMPLE_SITEMAP_BLUEPRINT'] = 'test_sitemap' app.config['SIMPLE_SITEMAP_URL'] = '/test_sitemap.xml' app.config['SIMPLE_SITEMAP_PATHS'] = { '/this_is_a_test': {'lastmod': '2017-04-24'} } @app.route('/hello') def hello(): return 'Hello' # assert lazy initialization extension.init_app(app) client = app.test_client() # # Final Test Setup #################################################################### #################################################################### # Cláusula que Permite testar manualmente o app com `python tests.py` # if __name__ == '__main__': app.run(debug=True) # # acesse localhost:5000/test_sitemap.xml #################################################################### #################################################################### # Agora sim os testes que serão executados com `py.test tests.py -v` # def test_sitemap_uses_custom_url(): response = client.get('/test_sitemap.xml') assert response.status_code == 200 def test_generated_sitemap_xml_is_valid(): response = client.get('/test_sitemap.xml') xml = response.data.decode('utf-8') result = xmltodict.parse(xml) assert 'urlset' in result # rules are Ordered assert result['urlset']['url'][0]['loc'] == '/test_sitemap.xml' assert result['urlset']['url'][1]['loc'] == '/hello' assert result['urlset']['url'][2]['loc'] == '/this_is_a_test' # # Ao terminar o tutorial reescreva esses testes usando fixtures :) # E é claro poderá adicionar mais testes! ###################################################################

Para que os testes acima sejam executados precisamos instalar algumas dependencias portanto o requirements-test.txt precisa deste conteúdo:

flask pytest xmltodict --editable .

NOTE: ao usar --editable . no arquivo de requirements você faz com que a extensão seja auto instalada em modo de edição desta forma executamos apenas pip install -r requirements-test.txt e o pip se encarrega de rodar o python setup.py develop.

Vamos então começar a desenvolver editando o front-end da extensão que será escrito no template: flask_simple_sitemap/templates/sitemap.xml este template espera um dict paths chaveado pela location e contento sitemap tag names em seu valor. exemplo paths = {'/artigos': {'lastmod': '2017-04-24'}, ...}

<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> {% for loc, data in paths.items() %} <url> <loc>{{loc|safe}}</loc> {% for tag_name, value in data.items() %} <{{tag_name}}>{{value}}</{{tag_name}}> {% endfor %} </url> {% endfor %} </urlset>

E então finalmente escreveremos a classe base da extensão no arquivo flask_simple_sitemap/base.py

Lembrando de algumas boas práticas:

  • Prover um método de inicialização lazy com a assinatura init_app(self, app)
  • Impedir registro em duplicidade e inserir um registro no app.extensions
  • Ao adicionar rotas sempre usar Blueprints e namespaces (o Blueprint já se encarrega do namespace nas urls)
  • Configs devem sempre ter um prefixo, faremos isso com o get_namespace que aprendemos na parte 1

NOTE: Leia atentamente os comentários e docstrings do código abaixo.

# coding: utf-8 from flask import Blueprint, render_template, make_response class SimpleSitemap(object): "Extensão Flask para publicação de sitemap" def __init__(self, app=None): """Define valores padrão para a extensão e caso o `app` seja informado efetua a inicialização imeditatamente caso o `app` não seja passado então a inicialização deverá ser feita depois (`lazy`) """ self.config = { 'blueprint': 'simple_sitemap', 'url': '/sitemap.xml', 'paths': {} } self.app = None # indica uma extensão não inicializada if app is not None: self.init_app(app) def init_app(self, app): """Método que Inicializa a extensão e pode ser chamado de forma `lazy`. É interessante que este método seja apenas o `entry point` da extensão e que todas as operações de inicialização sejam feitas em métodos auxiliares privados para melhor organização e manutenção do código. """ self._register(app) self._load_config() self._register_view() def _register(self, app): """De acordo com as boas práticas para extensões devemos checar se a extensão já foi inicializada e então falhar explicitamente caso seja verdadeiro. Se tudo estiver ok, então registramos o app.extensions e o self.app """ if not hasattr(app, 'extensions'): app.extensions = {} if 'simple_sitemap' in app.extensions: raise RuntimeError("Flask extension already initialized") # se tudo está ok! então registramos a extensão no app.extensions! app.extensions['simple_sitemap'] = self # Marcamos esta extensão como inicializada self.app = app def _load_config(self): """Carrega todas as variaveis de config que tenham o prefixo `SIMPLE_SITEMAP_` Por exemplo, se no config estiver especificado: SIMPLE_SITEMAP_URL = '/sitemap.xml' Podemos acessar dentro da extensão da seguinte maneira: self.config['url'] e isto é possível por causa do `get_namespace` do Flask utilizado abaixo. """ self.config.update( self.app.config.get_namespace( namespace='SIMPLE_SITEMAP_', lowercase=True, trim_namespace=True ) ) def _register_view(self): """aqui registramos o blueprint contendo a rota `/sitemap.xml`""" self.blueprint = Blueprint( # O nome do blueprint deve ser unico # usaremos o valor informado em `SIMPLE_SITEMAP_BLUEPRINT` self.config['blueprint'], # Agora passamos o nome do módulo Python que o Blueprint # está localizado, o Flask usa isso para carregar os templates __name__, # informamos que a pasta de templates será a `templates` # já é a pasta default do Flask mas como a nossa extensão está # adicionando um arquivo na árvore de templates será necessário # informar template_folder='templates' ) # inserimos a rota atráves do método `add_url_rule` pois fica # esteticamente mais bonito do que usar @self.blueprint.route() self.blueprint.add_url_rule( self.config['url'], # /sitemap.xml é o default endpoint='sitemap', view_func=self.sitemap_view, # usamos outro método como view methods=['GET'] ) # agora só falta registar o blueprint na app self.app.register_blueprint(self.blueprint) @property def paths(self): """Cria a lista de URLs que será adicionada ao sitemap. Esta property será executada apenas quando a URL `/sitemap.xml` for requisitada É interessante ter este método seja público pois permite que seja sobrescrito e é neste método que vamos misturar as URLs especificadas no config com as urls extraidas do roteamento do Flask (Werkzeug URL Rules). Para carregar URLs dinâmicamente (de bancos de dados) o usuário da extensão poderá sobrescrever este método ou contribur com o `SIMPLE_SITEMAP_PATHS` Como não queremos que exista duplicação de URLs usamos um dict onde a chave é a url e o valor é um dicionário completando os dados ex: app.config['SIMPLE_SITEMAP_PATHS'] = { '/artigos': { 'lastmod': '2017-01-01' }, ... } """ paths = {} # 1) Primeiro extraimos todas as URLs registradas na app for rule in self.app.url_map.iter_rules(): # Adicionamos apenas GET que não receba argumentos if 'GET' in rule.methods and len(rule.arguments) == 0: # para urls que não contém `lastmod` inicializamos com # um dicionário vazio paths[rule.rule] = {} # caso existam URLs que recebam argumentos então deverão ser carregadas # de forma dinâmica pelo usuário da extensão # faremos isso na hora de usar essa extensão no CMS de notícias. # 2) Agora carregamos URLs informadas na config # isso é fácil pois já temos o config carregado no _load_config paths.update(self.config['paths']) # 3) Precisamos sempre retornar o `paths` neste método pois isso permite # que ele seja sobrescrito com o uso de super(....) return paths def sitemap_view(self): "Esta é a view exposta pela url `/sitemap.xml`" # geramos o XML através da renderização do template `sitemap.xml` sitemap_xml = render_template('sitemap.xml', paths=self.paths) response = make_response(sitemap_xml) response.headers['Content-Type'] = 'application/xml' return response

NOTE: Neste exemplo usamos um método de desenvolvimento muito legal que eu chamo de:
ITF (Important Things First) onde Arquitetura, Documentação, Testes e Front End (e protótipos) são muito mais importantes do que a implementação de back end em si.
Assumimos que caso a nossa implementação seja alterada os conceitos anteriores se mantém integros com a proposta do produto.
Ordem de prioridade no projeto: 1) Definimos a arquitetura 2) Escrevemos documentação 3) Escrevemos testes 4) Implementamos front end (e protótipo) 5) back end é o menos importante do ponto de vista do produto e por isso ficou para o final! :)

O código da extensão etá disponível em http://github.com/rochacbruno/flask_simple_sitemap

Usando a extensão em nosso CMS de notícias

Agora vem a melhor parte, usar a extensão recém criada em nosso projeto existente.

O repositório do CMS está no github Precisamos do MongoDB em execução e a forma mais fácil é através do docker

➤ docker run -d -p 27017:27017 mongo

Se preferir utilize uma instância do MongoDB instalada localmente ou um Mongo As a Service.

NOTE: O modo de execução acima é efemero e não persiste os dados, para persistir use -v $PWD/etc/mongodata:/data/db.

Agora que o Mongo está rodando execute o nosso CMS.

Obtendo, instalando e executando:

➤ git clone -b extended --single-branch https://github.com/rochacbruno/wtf.git extended cd wtf # adicione nossa extensao nos requirements do CMS # sim eu publiquei no PyPI, mas se preferir instale a partir do fonte que vc escreveu echo 'flask_simple_sitemap' >> requirements.txt # activate a virtualenv pip install -r requirements.txt # execute python run.py

Agora com o CMS executando acesse http://localhost:5000 e verá a seguinte tela:

Os detalhes dessa aplicação você deve ser lembrar pois estão nas partes 1, 2 e 3 deste tutorial.

Agora você pode se registrar novas notícias usando o link cadastro e precisará efetuar login e para isso deve se registrar como usuário do aplicativo.

Temos as seguintes urls publicads no CMS
  • '/' lista todas as noticias na home page
  • '/noticias/cadastro' exibe um formulário para incluir noticias
  • '/noticia/<id> acessa uma noticia especifica
  • '/admin' instancia do Flask Admin para adminstrar usuários e o banco de dados

Agora vamos incluir a extensão flask_simple_sitemap que criamos e adicionar as URLs das noticias dinâmicamente.

Edite o arquico wtf/news_app.py incluindo a extensão flask_simple_sitemap e também adicionando as URLs de todas as noticias que existirem no banco de dados.

# coding: utf-8 from os import path from flask import Flask from flask_bootstrap import Bootstrap from flask_security import Security, MongoEngineUserDatastore from flask_debugtoolbar import DebugToolbarExtension ############################################### # 1) importe a nossa nova extensão from flask_simple_sitemap import SimpleSitemap from .admin import configure_admin from .blueprints.noticias import noticias_blueprint from .db import db from .security_models import User, Role from .cache import cache ############################################## # 2) importe o model de Noticia from .models import Noticia def create_app(mode): instance_path = path.join( path.abspath(path.dirname(__file__)), "%s_instance" % mode ) app = Flask("wtf", instance_path=instance_path, instance_relative_config=True) app.config.from_object('wtf.default_settings') app.config.from_pyfile('config.cfg') app.config['MEDIA_ROOT'] = path.join( app.config.get('PROJECT_ROOT'), app.instance_path, app.config.get('MEDIA_FOLDER') ) app.register_blueprint(noticias_blueprint) Bootstrap(app) db.init_app(app) Security(app=app, datastore=MongoEngineUserDatastore(db, User, Role)) configure_admin(app) DebugToolbarExtension(app) cache.init_app(app) ############################################ # 3) Adicionane as noticias ao sitemap app.config['SIMPLE_SITEMAP_PATHS'] = { '/noticia/{0}'.format(noticia.id): {} # dict vazio mesmo por enquanto! for noticia in Noticia.objects.all() } ############################################ # 4) Inicialize a extensão SimpleSitemap sitemap = SimpleSitemap(app) return app

Agora execute o python run.py e acesse http://localhost:5000/sitemap.xml

Você verá o sitemap gerado incluindo as URLs das notícias cadastradas!

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>/noticia/58ffe998e138231eef84f9a7</loc> </url> <url> <loc>/noticias/cadastro</loc> </url> <url> <loc>/</loc> </url> ... # + um monte de URL do /admin aqui </urlset>

NOTE: Funcionou! legal! porém ainda não está bom. Existem algumas melhorias a serem feitas e vou deixar essas melhorias para você fazer!

Desafio What The Flask! 1) Melhore a geração de URLs do CMS

Você reparou que a URL das notícias está bem feia? /noticia/58ffe998e138231eef84f9a7 não é uma boa URL Para ficar mais simples no começo optamos por usar o id da notícia como URL mas isso não é uma boa prática e o pior é que isso introduz até mesmo problemas de segurança.

Você conseguer arrumar isso? transformando em: /noticia/titulo-da-noticia-aqui ?

Vou dar umas dicas:

Altere o Model:

  • Altere o model Noticia em: https://github.com/rochacbruno/wtf/blob/extended/wtf/models.py#L5.
  • Insira um novo campo para armazenar o slug da notícia, o valor será o título transformado para lowercase, espaços substituidos por -. Ex: para o título: 'Isto é uma notícia' será salvo o slug: 'isto-e-uma-noticia'.
  • Utilize o método save do MongoEngine ou se preferir use signals para gerar o slug da notícia.
  • Utilize o módulo awesome-slugify disponível no PyPI para criar o slug a partir do título.

Altere a view:

Altere as urls passadas ao SIMPLE_SITEMAP_PATHS usando o slug ao invés do id.

2) Adicione data de publicação nas notícias

Reparou que o sitemap está sem a data da notícia? adicione o campo modified ao model Noticia e faça com que ele salve a data de criação e/ou alteração da notícia.

Queremos algo como:

app.config['SIMPLE_SITEMAP_PATHS'] = { '/noticia/{0}'.format(noticia.slug): { 'lastmod': noticia.modified.strftime('%Y-%m-%d') } for noticia in Noticia.objects.all() }

Para gerar no sitemap:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>/noticia/titulo-da-noticia</loc> <lastmod>2017-04-25</lastmod> </url> ... 3) Crie uma opção de filtros na extensão simple_sitemap

Uma coisa chata também é o fato do sitemap.xml ter sido gerado com esse monte de URL indesejada. As URLs iniciadas com /admin por exemplo não precisam ir para o sitemap.xml.

Implemente esta funcionalidade a extensão:

DICA: Use regex import re

app.config['SIMPLE_SITEMAP_EXCLUDE'] = [ # urls que derem match com estes filtros não serão adicionadas ao sitemap '^/admin/.*' ]

DESAFIO: Após implementar as melhorias inclua nos comentários um link para a sua solução, pode ser um fork dos repositórios ou até mesmo um link para gist ou pastebin (enviarei uns adesivos de Flask para quem completar o desafio!)

O diff com as alterações realizadas no CMS encontra-se no github.com/rochacbruno/wtf
A versão final da extensão SimpleSitemap está no github.com/rochacbruno/flask_simple_sitemap
A versão final do CMS app está no github.com/rochacbruno/wtf

Se você está a procura de uma extensão para sitemap para uso em produção aconselho a flask_sitemap

END: Sim chegamos ao fim desta quarta parte da série What The Flask. Eu espero que você tenha aproveitado as dicas aqui mencionadas. Nas próximas partes iremos efetuar o deploy de aplicativos Flask. Acompanhe o PythonClub, o meu site e meu twitter para ficar sabendo quando a próxima parte for publicada.

PUBLICIDADE: Iniciarei um curso online de Python e Flask, para iniciantes abordando com muito mais detalhes e exemplos práticos os temas desta série de artigos e muitas outras coisas envolvendo Python e Flask, o curso será oferecido no CursoDePython.com.br, ainda não tenho detalhes especificos sobre o valor do curso, mas garanto que será um preço justo e acessível. Caso você tenha interesse por favor preencha este formulário pois dependendo da quantidade de pessoas interessadas o curso sairá mais rapidamente.

PUBLICIDADE 2: Também estou escrevendo um livro de receitas Flask CookBook através da plataforma LeanPub, caso tenha interesse por favor preenche o formulário na página do livro

PUBLICIDADE 3: Inscreva-se no meu novo canal de tutoriais

Muito obrigado e aguardo seu feedback com dúvidas, sugestões, correções etc na caixa de comentários abaixo.

Abraço! "Python é vida!"

Categories: FLOSS Project Planets

Evolving Web: Contribute by Saying #DrupalThanks at DrupalCon Baltimore

Planet Drupal - Wed, 2017-04-26 06:41

Drupal and the Drupal community are driven by volunteer contributions. There are many great ways to contribute that don't involve writing a single line of code. You could report a bug, edit a documentation page, test a new Drupal 8 feature, help a beginner on a forum, or help organize a meetup. Despite this, many Drupalers are shy about diving in.

After some reflection, the Evolving Web team realized that there's one way to contribute to the community that's really simple: thanking someone who has helped you. Someone who built a module you used, helped you in the issue queue, or has done a presentation you liked. It doesn't matter if you do it privately, publicly on Twitter, or whether you're Dries or a total beginner. 

As part of Evolving Web's "Drupal Love" sponsorship at DrupalCon Baltimore, we're hoping to encourage spontaneous expressions of gratitude by handing out 1,000 flowers, with a simple request: Give this flower to someone at DrupalCon Baltimore who helped you in some way. Spread #DrupalThanks.

It will make both you and them feel great, encourage contributions, prevent burnout, and maybe even start a virtuous cycle. Thanking somebody costs nothing, yet can mean so much.

Evolving Web will be handing out several prizes to both givers and receivers of public thanks, which include a day of free Drupal Training, a subsidized trip to attend DrupalCamp Montréal this June 15-18, and flower bouquets delivered to your home or office.

To participate, just thank someone publicly on Twitter, like this:

or like this:

or like this:

Or if you'd prefer to do it offline, come by our Evolving Web's booth and tell us who you are thanking.

 

+ more awesome articles by Evolving Web
Categories: FLOSS Project Planets

Wesley Chun: Exporting a Google Sheet spreadsheet as CSV

Planet Python - Wed, 2017-04-26 06:12
IntroductionToday, we'll follow-up to my earlier post on the Google Sheets API and multiple posts (first, secondthird) on the Google Drive API by answering one common question: How do you download a Google Sheets spreadsheet as a CSV file? The "FAQ"ness of the question itself as well as various versions of Google APIs has led to many similar StackOverflow questions: one, two, three, four, five, just to list a few. Let's answer this question definitively and walk through a Python code sample that does exactly that. The main assumption is that you have a Google Sheet file in your Google Drive named "inventory".

Choosing the right APIUpon first glance, developers may think the Google Sheets API is the one to use. Unfortunately that isn't the case. The Sheets API is the one to use for spreadsheet-oriented operations, such as inserting data, reading spreadsheet rows, managing individual tab/sheets within a spreadsheet, cell formatting, creating charts, adding pivot tables, etc., It isn't meant to perform file-based requests like exporting a Sheet in CSV (comma-separated values) format. For file-oriented operations with a Google Sheet, you would use the Google Drive API.

Using the Google Drive APIAs mentioned earlier, Google Drive features numerous API scopes of authorization. As usual, we always recommend you use the most restrictive scope possible that allows your app to do its work. You'll request fewer permissions from your users (which makes them happier), and it also makes your app more secure, possibly preventing modifying, destroying, or corrupting data, or perhaps inadvertently going over quotas. Since we're only exporting a Google Sheets file from Google Drive, the only scope we need is:
  • 'https://www.googleapis.com/auth/drive.readonly' — Read-only access to file content or metadata
The earlier post I wrote on the Google Drive API featured sample code that exported an uploaded Google Docs file as PDF and download that from Drive. This post will not only feature a change to exporting a Google Sheets file in CSV format, but also demonstrate one additional feature of the Drive API: querying

Since we've fully covered the authorization boilerplate fully in earlier posts and videos, we're going to skip that here and jump right to the action, creating of a service endpoint to Drive. The API name is (of course 'drive', and the current version of the API is 3, so use the string 'v3' in this call to the apiclient.discovey.build() function:

DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

Query and export files from Google DriveWhile unnecessary, we'll create a few string constants representing the filename, source and destination file MIME types to make the code easier to understand:
FILENAME = 'inventory'
SRC_MIMETYPE = 'application/vnd.google-apps.spreadsheet'
DST_MIMETYPE = 'text/csv'
In this simple example, we're only going to export one Google Sheets file as CSV, arbitrarily choosing a file named, "inventory." So to perform the query, you need both the filename and its MIME type, "application/vnd.google-apps.spreadsheet". Query components are conjoined with the "and" keyword, so your query string will look like this: q='name="%s" and mimeType="%s"' % (FILENAME, SRC_MIMETYPE).

Since there may be more than one Google Sheets file named 'inventory". we opt for newest one and thus need to sort all matching files in descending order of last modification time then name if "mtime"s are identical via an "order by" clause: orderBy='modifiedTime desc,name'. Here is the complete call to DRIVE.files().list() to issue the query:
files = DRIVE.files().list(
q='name="%s" and mimeType="%s"' % (FILENAME, SRC_MIMETYPE),
orderBy='modifiedTime desc,name').execute().get('files', [])
If any files match, the payload will contain a 'files' key, else we default to an empty list and display to the user on the last line that no files were found. Otherwise, grab the first match, the most recently-modified 'inventory' file, create a suitable CSV filename from it, and change all spaces to underscores:

fn = '%s.csv' % os.path.splitext(files[0]['name'].replace(' ', '_'))[0]

The final Drive API call requests an export of 'inventory' as a CSV file, and if successful, the downloaded data is written with the filename above. In either case, the user is notified of success or failure of the export:
data = DRIVE.files().export(fileId=files[0]['id'], mimeType=DST_MIMETYPE).execute()
if data:
with open(fn, 'wb') as f:
f.write(data)
print('DONE')
else:
print('ERROR (could not download file)')
Note that if downloading as CSV, the Drive API only exports of the first sheet in a Sheets file... you won't get any others. However, it does support 3 other download formats that will get you all the sheets.

If you create a Sheets file named 'inventory', run the script, grant the script access to your Google Drive (via the OAuth2 prompt that pops up in the browser), and then you should get output that looks like this:
$ python drive_sheets_csv_export.py # or python3
Exporting "inventory" as "inventory.csv"... DONEConclusionBelow is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!). If I were to divide the script into 4 major sections, they would be:
  • Get creds & build Google Drive service endpoint
  • Source and destination file info
  • Query Google Drive for matching files
  • Export most recent matching Sheets file as CSV

Here's the code itself:
from __future__ import print_function
import os

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/drive.readonly'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))

FILENAME = 'inventory'
SRC_MIMETYPE = 'application/vnd.google-apps.spreadsheet'
DST_MIMETYPE = 'text/csv'

files = DRIVE.files().list(
q='name="%s" and mimeType="%s"' % (FILENAME, SRC_MIMETYPE),
orderBy='modifiedTime desc,name').execute().get('files', [])

if files:
fn = '%s.csv' % os.path.splitext(files[0]['name'].replace(' ', '_'))[0]
print('Exporting "%s" as "%s"... ' % (files[0]['name'], fn), end='')
data = DRIVE.files().export(fileId=files[0]['id'], mimeType=DST_MIMETYPE).execute()
if data:
with open(fn, 'wb') as f:
f.write(data)
print('DONE')
else:
print('ERROR (could not download file)')
else:
print('!!! ERROR: File not found')
As with our other code samples, you can now customize for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs. Hope this helps answer yet another frequently asked question!
Categories: FLOSS Project Planets

Sven Hoexter: Chrome 58 ignores commonName in certificates

Planet Debian - Wed, 2017-04-26 06:08

People using Chrome might have already noticed that some internal certificates created without a SubjectAlternativeName extension fail to verify. Finally the Google Chrome team stepped forward, and after only 17 years of having SubjectAlternativeName as the place for FQDNs to verify as valid for a certificate, they started to ignore the commonName. See also https://www.chromestatus.com/feature/4981025180483584.

Currently Debian/stretch still has Chromium 57 but Chromium 58 is already in unstable. So some more people might notice this change soon. I hope that everyone who maintains some broken internal scripting to maintain internal CAs now re-reads the OpenSSL Cookbook to finally fix this stuff. In general I recommend to base your internal CA scripting on easy-rsa to avoid making every mistake in certificate management on your own.

Categories: FLOSS Project Planets

Dropsolid: Drupal 8 migration strategies

Planet Drupal - Wed, 2017-04-26 03:00
26 Apr Drupal 8 migration strategies Kevin VB

Content is always one of the most important parts of a website. After all, the internet was designed for information sharing. As a result, upgrading to a new CMS implies the migration of content in some form or another. Taking this into account in the early stages is crucial when considering and preparing a CMS upgrade.
The Drupal community is very aware of content migration as a key success factor. Therefore, Drupal’s latest version (D8) has the
migrate module included in its core. This key functionality allows you to upgrade from an older Drupal version to Drupal 8, using a few simple configuration steps. Below, I will explain how this works and put forward a few alternatives for custom migrations.

 

Why should you migrate to Drupal 8?

Site speed is not only important for SEO; it also affects your visitors’ browsing time and exit rate. Drupal 8 comes with an improved caching system that makes Drupal fly - all while taking into account content modification. There are no more endless waits for caches to invalidate, thanks to the real-time cache invalidation using cache tags.
Another reason for migration is the Drupal community. Plenty of features are available to integrate in your own site for free, which in turn enables you to spend time and money on other things. The community also keeps an eye on continuous improvements to the existing code. Drupal 8 is a great example of this, with its foundations in the Symfony2 framework. Everything in D8 has been standardised in such a way that maintenance is a lot easier and time-effective.
Let there be no doubt that migrating to Drupal 8 is an excellent long-term move!

 

How exactly should I migrate to Drupal 8?

You can use the Drupal migrate module that is included in core to upgrade from an older Drupal version to Drupal 8. Make sure to install and enable required modules first.
An example: if your site uses special field types, those modules should also be installed in your new Drupal 8 website. When you’re done configuring your site, you just need to enable the following modules:

  • Migrate
  • Migrate Drupal
  • Migrate Drupal UI

This last module will direct you to a configuration page, where you can start the actual migration. Simply enter the database information from your existing Drupal site and let Drupal review the upgrade.

The review will give you a list of available upgrade paths, next to a list of modules that are currently missing. If you’re happy about the review, you can choose to start the upgrade. Drupal will start importing content, users and taxonomies into your Drupal 8 website. Be aware that a rollback mechanism through the UI of Drupal is not available at this time. Since the Drupal core migrate is built to support a certain number of cases, it is possible that your site is too complicated to import correctly with the Migrate Drupal module. Sometimes, writing a customised migration is a better approach.
 

How to write a customized migration?

In most cases, the Migrate Drupal module will result in a reinstall of your Drupal website because some parts have been imported in the wrong way. You can opt to play things safe and write the migration yourself.
Writing a migration in Drupal 8 is done with the Migrate Plus module. This module allows you to create a new Migration entity. Those entities are created in YAML.

# Migration configuration for News content. id: news_node label: News Content Type migration_group: demo_news source: plugin: news_node destination: plugin: entity:node process: type: plugin: default_value default_value: news langcode: plugin: default_value source: language_code default_value: nl title: post_title field_title: post_title path: path field_tags: plugin: migration migration: - news_terms source: tags_terms migration_dependencies: required: - news_terms

Example of a Migration entity: migrate_plus.migration.news_node.yml

Each migration entity can belong to a Migration Group entity and is defined in YAML with the key ‘migrate_group’. A whole group of migrations can be imported or rolled back at once with drush by installing the Migrate Tools module.

  • drush mi --group=”demo_news” Import all migration in group demo_news
  • drush mr --group=”demo_news” Rollback all migrations in group demo_news

 

The main key of a migration is a Drupal 8 plugin that tells the migration where the source information comes from. There are plenty of base source plugins available for Drupal 8.

  • SqlBase - in Drupal Core: lets you migrate from an SQL source.
  • URL - in Migrate Plus: lets you migrate from a URL which can return JSON, XML, SOAP.
  • CSV - in Migrate Source CSV: lets you migrate from a CSV source file.
  • Spreadsheet - in Migrate Spreadsheet: lets you migrate from a csv, xls or xlsx source file.

If this does not suffice, you can start from your own source that extends from the SourcePluginBase class.

We extended the SqlBase source for our news_node source plugin.

public function query() { $query = $this->select('post', 'p'); $query->fields('p', [ 'ID', 'post_title', 'post_name', 'post_date', ]); $query->condition('p.type', 'news'); return $query; }

Query function in news_node source.

The query function returns a Select object with the information needed during the migration. This object will be the source for our migration.
Next we need to tell the migration which fields we want to map to the migration. This is done with the fields method.

public function fields() { $fields = [ 'post_title' => $this->t('The Post Node title'), 'post_date' => $this->t('The Post creation time'), // ... ]; return $fields; }

In Drupal 7 we used prepareRow to provide field information that couldn’t be selected with a single query. In Drupal 8 this can be done with the prepareRow method. In our example we fetch a teaser image and then add the file ID and file alt to the migration source.

public function prepareRow(Row $row) { // Find related teaser attachment image. $file = $this->getTeaserImage($content_id); // Set a new property file_id. $row->setSourceProperty('file_id', $file['id']); $row->setSourceProperty('file_alt', $file['alt']); return parent::prepareRow($row); }

Add extra information to the migration in preparerow.

When we go back to the YAML confirmation of our migration entity, we see that there is also a destination key configured. In most cases this will be an entity:entity_type destination plugin. Migrate will then automatically create entities of the configured type. In our example, new nodes will be created. If needed, you can also simply create a new destination plugin, which performs extra actions during the import function.

 

The progress key in our configuration defines the field value mapping. It contains a mapping of keys and values where the key is the Drupal field name and the value is the source field name. In some cases, like ‘type’ or ‘language’, we use a default_value plugin which allows us to set the value of the field to a fixed value. In our example, we are creating new nodes of type news in Dutch.

In some cases, the source value comes from another migration. In our example the value of ‘field_tags’ comes from another migration, this is defined by using the ‘migration’ plugin and then specify the migration(s) in which the value is migrated. Whenever such migration dependent fields are presents an extra key ‘migration dependencies’ is necessary. This is an array of migrations which needs to run first.


I hope this post has helped you to provide some insight in migrating your website from D7 to D8! As always, you can reach out to me and the rest of the team via our website.

Categories: FLOSS Project Planets

Gocept Weblog: Last call for take off to the Python 3 wonderland

Planet Python - Wed, 2017-04-26 02:56

We are approaching the Zope 2 Resurrection Sprint and hope that all those who are willing to help earl Zope II on his endeavor to port his realm have already prepared there horses and packed the necessary equipment to arrive in Halle (Saale), Germany.

To help with the preparations we have set up some means of communication:

Etherpad

In the Etherpad we have collected a fundamental set of obstacles that the immigration authority of Python 3 wonderland send to earl Zope II via a mounted messenger. If there are additional problems we need to solve with the immigration or other authorities, feel free to point those out in the pad.

IRC Channel

During the sprint we will have an owl waiting for messages in the #sprint channel on irc.freenode.net, so additional information and questions can be placed there.

General Schedule

In general the gates of the gocept manor in Halle (Saale) are open from 8:00 till 18:00 during the sprint for the squires to help earl Zope II. There will be some refreshments in the morning (8:00 – 9:00) and during lunch time (12:00 – 13:00) in order to keep everyone happy and content.

Apart from that, there will be some fixed points in time to meet:

  • Monday 2017-05-01
    • 19:00 CEST, pre-sprint get-together for early arrivals at Anny Kilkenny. Attention: There will be a bigger political demonstration in Halle which might impact the arrival here, take that into consideration.
  • Tuesday 2017-05-02
    • 9:00 CEST, official welcome and sprint planning afterwards.
    • 16:30-17:30 CEST, Discussion: TBD
    • 18:00 CEST, guided tour through the city of Halle, meeting point
    • 19:30 CEST, dinner and get-together at Wenzels Bierstuben, location, separate bills
  • Wednesday 2017-05-03
    • 9:00 CEST, daily meeting and review
    • 16:30-18:00 CEST, Discussion: TDB
    • 19:00 CEST, BBQ evening in the lovely garden at gocept manor
  • Thursday 2017-05-04
  • Friday 2017-05-05
    • 9:00 CEST, daily meeting and review
    • 13:00 CEST, sprint closing session with review and possibility to present lightning talks of your projects.

We are looking forward to the sprint and hope to overcome the remaining migration problems of earl Zope II.


Categories: FLOSS Project Planets

Piergiorgio Lucidi: Apache ManifoldCF 2.7: New UX and SharePoint 2016 support

Planet Apache - Wed, 2017-04-26 02:30

We are proud to announce that Apache ManifoldCF 2.7 was released just some days ago.

Categories: FLOSS Project Planets

Russ Allbery: Review: Zero Bugs and Program Faster

Planet Debian - Tue, 2017-04-25 23:17

Review: Zero Bugs and Program Faster, by Kate Thompson

Publisher: Kate Thompson Copyright: 2015 Printing: December 2016 ISBN: 0-9961933-0-8 Format: Trade paperback Pages: 169

Zero Bugs and Program Faster is another book I read for the engineering book club at work. Unlike a lot of the previous entries, I'd never heard about it before getting it for the book club and had no idea what to expect. What I got was a lavishly-illustrated book full of quirky stories and unusual turns of presentation on the general subject of avoiding bugs in code. Unfortunately, it's not a very deep examination of the topic. All that an experienced programmer is likely to get out of this book is the creative storytelling and the occasionally memorable illustration.

I knew that this may not be a book aimed at me when I read the first paragraph:

If two books in a bookstore teach the same thing, I take the shorter one. Why waste time reading a 500-page book if I can learn the same in 100 pages? In this book, I kept things clear and brief, so you can ingest the information quickly.

It's a nice thought, but there are usually reasons why the 500-page book has 400 more pages, and those are the things I was missing here. Thompson skims over the top of every topic. There's a bit here on compiler warnings, a bit on testing, a bit on pair programming, and a bit on code review, but they're all in extremely short chapters, usually with some space taken up with an unusual twist of framing. This doesn't leave time to dive deep into any topic. You won't be bored by this book, but most of what you'll earn is "this concept exists." And if you've been programming for a while, you probably know that already.

I learned during the work discussion that this was originally a blog and the chapters are repurposed from blog posts. I probably should have guessed at that, since that's exactly what the book feels like. It's a rare chapter that's longer than a couple of pages, including the room for the illustrations.

The illustrations, I must say, are the best part of the book. They're are a ton of them, sometimes just serving the purpose of stock photography to illustrate a point (usually with a slightly witty caption), sometimes a line drawing to illustrate some point about code, sometimes an illustrated mnemonic for the point of that chapter. The book isn't available in a Kindle edition precisely because including the illustrations properly is difficult to do (per its Amazon page).

As short as this book is, only about two-thirds of it is chapters about programming. The rest is code examples (not that the chapters themselves were light on code examples). Thompson introduces these with:

One of the best ways to improve your programming is to look at examples of code written by others. On the next pages you will find some code I like. You don't need to read all of it: take the parts you like, and skip the parts you don't. I like it all.

I agree with this sentiment: reading other people's code is quite valuable. But it's also very easy to do, given a world full of free software and GitHub repositories. When reading a book, I'm looking for additional insight from the author. If the code examples are beautiful enough to warrant printing here, there presumably is some reason why. But Thompson rarely does more than hint at the reason for inclusion. There is some commentary on the code examples, but it's mostly just a discussion of their origin. I wanted to know why Thompson found each piece of code beautiful, as difficult as that may be to describe. Without that commentary, it's just an eclectic collection of code fragments, some from real systems and some from various bits of stunt programming (obfuscation contests, joke languages, and the like).

The work book club discussion showed that I wasn't the only person disappointed by this book, but some people did like it for its unique voice. I don't recommend it, but if it sounds at all interesting, you may want to look over the corresponding web site to get a feel for the style and see if this is something you might enjoy more than I did.

Rating: 5 out of 10

Categories: FLOSS Project Planets

Bryan Pendleton: GBooks, 15 years on

Planet Apache - Tue, 2017-04-25 22:36

Three perspectives:

  • How Google Book Search Got LostTwo things happened to Google Books on the way from moonshot vision to mundane reality. Soon after launch, it quickly fell from the idealistic ether into a legal bog, as authors fought Google’s right to index copyrighted works and publishers maneuvered to protect their industry from being Napsterized. A decade-long legal battle followed — one that finally ended last year, when the US Supreme Court turned down an appeal by the Authors Guild and definitively lifted the legal cloud that had so long hovered over Google’s book-related ambitions.

    But in that time, another change had come over Google Books, one that’s not all that unusual for institutions and people who get caught up in decade-long legal battles: It lost its drive and ambition.

    ...

    But Google took away a lesson that helped it immeasurably as it grew and gained power: Engineering is great, but it’s not the answer to all problems. Sometimes you have to play politics, too — consult stakeholders, line up allies, compromise with rivals. As a result, Google assembled a crew of lobbyists and lawyers and approached other similar challenges — like navigating YouTube’s rights maze — with greater care and better results.

  • Torching the Modern-Day Library of AlexandriaWhat happened was complicated but how it started was simple: Google did that thing where you ask for forgiveness rather than permission, and forgiveness was not forthcoming. Upon hearing that Google was taking millions of books out of libraries, scanning them, and returning them as if nothing had happened, authors and publishers filed suit against the company, alleging, as the authors put it simply in their initial complaint, “massive copyright infringement.”

    ...

    Amazon, for its part, worried that the settlement allowed Google to set up a bookstore that no one else could. Anyone else who wanted to sell out-of-print books, they argued, would have to clear rights on a book-by-book basis, which was as good as impossible, whereas the class action agreement gave Google a license to all of the books at once.

    This objection got the attention of the Justice Department, in particular the Antitrust division, who began investigating the settlement. In a statement filed with the court, the DOJ argued that the settlement would give Google a de facto monopoly on out-of-print books. That’s because for Google’s competitors to get the same rights to those books, they’d basically have to go through the exact same bizarre process: scan them en masse, get sued in a class action, and try to settle.

    ...

    It was strange to me, the idea that somewhere at Google there is a database containing 25-million books and nobody is allowed to read them. It’s like that scene at the end of the first Indiana Jones movie where they put the Ark of the Covenant back on a shelf somewhere, lost in the chaos of a vast warehouse. It’s there. The books are there. People have been trying to build a library like this for ages—to do so, they’ve said, would be to erect one of the great humanitarian artifacts of all time—and here we’ve done the work to make it real and we were about to give it to the world and now, instead, it’s 50 or 60 petabytes on disk, and the only people who can see it are half a dozen engineers on the project who happen to have access because they’re the ones responsible for locking it up.

  • Why Google Books Deserves Better Than These ObituariesUnfortunately, the copyright case over Google Books morphed into something larger. It became a vehicle for anxieties over how the digital era has undermined authors on a financial and cultural level. Those concerns are legitimate, but scapegoating Google Books for these fears was misguided.

    Meanwhile, groups like the Authors Guild continue to celebrate the collapse of the settlement even though no other options have emerged to replicate its potential benefits. Those benefits would have included a new market for digital copies of old books, and a solution to the problem of "orphan works"—books whose authors cannot be located that are out-of-print but still under copyright protection. Instead, there is stasis.

    ...

    But if Google is to dispel the recent mutterings about the project's decline, it must do more to raise the profile of Google Books, and offer some assurances about how it will ensure the collection—which, recall, is nothing less than the history of human knowledge—will survive. Ideally, the company should create a trust or foundation to manage it so as to ensure it endures no matter what corporate changes come at Google.

    Meanwhile, it's time for Google Books opponents to acknowledge the astonishing thing Google has built. Critics like the former head of Harvard libraries, Robert Darnton, have long suggested some university or public consortium can replicate the project. But today it's clearer than ever this is just a pipeline, and no one will muster the money, energy, and technology to do what Google did also over again.

Categories: FLOSS Project Planets

Dirk Eddelbuettel: RcppTOML 0.1.3

Planet Debian - Tue, 2017-04-25 21:01

A new bug fix release of RcppTOML arrived on CRAN today. Table arrays were (wrongly) not allowing for nesting; a simply recursion fix addresses this.

RcppTOML brings TOML to R. TOML is a file format that is most suitable for configurations, as it is meant to be edited by humans but read by computers. It emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. On small typos you get parse errors, rather than silently corrupted garbage. Much preferable to any and all of XML, JSON or YAML -- though sadly these may be too ubiquitous now. TOML is making good inroads with newer and more flexible projects such as the Hugo static blog compiler, or the Cargo system of Crates (aka "packages") for the Rust language.

Changes in version 0.1.3 (2017-04-25)
  • Nested TableArray types are now correctly handled (#16)

Courtesy of CRANberries, there is a diffstat report for this release.

More information is on the RcppTOML page page. Issues and bugreports should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets

Justin Mason: Links for 2017-04-25

Planet Apache - Tue, 2017-04-25 19:58
  • Ireland’s Content Pool

    Bring your content to life with our free resource for positive tourism related purposes. Our image, video and copy collections show people, landscapes and the Irish lifestyle across a range of experiences including festivals, activities, cities, rural life and food. Interesting idea — but the licensing terms aren’t 100% clear. This would have been much easier if it was just CC licensed!

    (tags: open-data licensing ireland tourism via:damienmulley landscapes photos pictures content failte-ireland)

  • Here’s Why Juicero’s Press is So Expensive – Bolt Blog

    Our usual advice to hardware founders is to focus on getting a product to market to test the core assumptions on actual target customers, and then iterate. Instead, Juicero spent $120M over two years to build a complex supply chain and perfectly engineered product that is too expensive for their target demographic. Imagine a world where Juicero raised only $10M and built a product subject to significant constraints. Maybe the Press wouldn’t be so perfectly engineered but it might have a fewer features and cost a fraction of the original $699. Or maybe with a more iterative approach, they would have quickly found that customers vary greatly in their juice consumption patterns, and would have chosen a per-pack pricing model rather than one-size-fits-all $35/week subscription. Suddenly Juicero is incredibly compelling as a product offering, at least to this consumer.

    (tags: juicero design electronics hardware products startups engineering teardowns)

  • AWS Greengrass

    AWS Greengrass is software that lets you run local compute, messaging & data caching for connected devices in a secure way. With AWS Greengrass, connected devices can run AWS Lambda functions, keep device data in sync, and communicate with other devices securely – even when not connected to the Internet. Using AWS Lambda, Greengrass ensures your IoT devices can respond quickly to local events, operate with intermittent connections, and minimize the cost of transmitting IoT data to the cloud. AWS Greengrass seamlessly extends AWS to devices so they can act locally on the data they generate, while still using the cloud for management, analytics, and durable storage. With Greengrass, you can use familiar languages and programming models to create and test your device software in the cloud, and then deploy it to your devices. AWS Greengrass can be programmed to filter device data and only transmit necessary information back to the cloud. AWS Greengrass authenticates and encrypts device data at all points of connection using AWS IoT’s security and access management capabilities. This way data is never exchanged between devices when they communicate with each other and the cloud without proven identity.

    (tags: aws cloud iot lambda devices offline synchronization architecture)

  • Immunotherapy Pioneer James Allison Has Unfinished Business with Cancer – MIT Technology Review

    On the discovery and history of ipilimumab (trade named Yervoy), one of the first immunotherapy drugs

    (tags: ipilimumab cancer yervoy immunotherapy medicine melanoma)

  • FactCheck: No, the reported side effects of the HPV vaccine do NOT outweigh the proven benefits

    The Journal FactCheck team take a shortcut through Regret.ie’s bullshit

    (tags: hpv antivaxxers gardasil safety vaccination health medicine fact-checking)

Categories: FLOSS Project Planets

Carl Chenet: Migrate Feed2tweet from 1.0 to 1.1

Planet Python - Tue, 2017-04-25 18:00

Feed2tweet 1.1, your RSS to Twitter bot, had a compatibility-breaking change: the format of the cache file changed because of a Python platform-dependent issue of one of the dependencies.

 

It is mandatory to execute the following steps in order to keep safe the timeline of your Twitter account (and maybe more important the timelines of your followers).

How to migrate Feed2tweet 1.0 to 1.1

We start by commenting the Feed2tweet entry in the system crontab:

# */10 * * * * feed2tweet feed2tweet -c /etc/feed2tweet/feed2tweet.ini

The next step is to update Feed2tweet:

# pip3 install feed2tweet --upgrade

As we may have an issue during the upgrade, we back up the cache file:

$ mkdir feed2tweet-backup $ cp /var/lib/feed2tweet/feed2tweet.db feed2tweet-backup/

Now we remove the Feed2tweet cache file:

$ rm -f /var/lib/feed2tweet/feed2tweet.db

So far so good, let’s regenerate the cache file by getting all then entries but NOT sending them to Twitter:

$ feed2tweet --populate-cache -c /etc/feed2tweet/feed2tweet.ini

Almost finished. We execute the dry run mode in order to check the result. You should not have any displayed entry, meaning you’re now ready to send the next ones to Twitter:

$ feed2tweet --dry-run -c /etc/feed2tweet/feed2tweet.ini

And of course to finish we uncomment the Feed2tweet line in the /etc/crontab file

*/10 * * * * feed2tweet feed2tweet -c /etc/feed2tweet/feed2tweet.ini

We’re all set! The new RSS entries of your feeds will be automatically posted to Twitter.

More information about Feed2tweet … and finally

You can help the Feed2tweet Bot by donating anything through Liberaypay (also possible with cryptocurrencies). That’s a big factor motivation

Categories: FLOSS Project Planets

Philip Semanchuk: Thanks, Science!

Planet Python - Tue, 2017-04-25 17:20

I took part in the Raleigh March for Science last Saturday. For the opportunity to learn about it, participate in it, photograph it, share it with you — oh, and also for, you know, being alive today — thanks, science!







Categories: FLOSS Project Planets

Cutelyst 1.6.0 released, to infinity and beyond!

Planet KDE - Tue, 2017-04-25 16:09

Once 1.5.0 was release I thought the next release would be a small one, it started with a bunch of bug fixes, Simon Wilper made a contribution to Utils::Sql, basically when things get out to production you find bugs, so there were tons of fixes to WSGI module.

Then TechEmpower benchmarks first preview for round 14 came out, Cutelyst performance was great, so I was planning to release 1.6.0 as it was but second preview fixed a bug that Cutelyst results were scaled up, so our performance was worse than on round 13, and that didn’t make sense since now it had jemalloc and a few other improvements.

Actually the results on the 40+HT core server were close to the one I did locally with a single thread.

Looking at the machine state it was clear that only a few (9) workers were running at the same time, I then decided to create an experimental connection balancer for threads. Basically the main thread accepts incoming connections and evenly pass them to each thread, this of course puts a new bottleneck on the main thread. Once the code was ready which end up improving other parts of WSGI I became aware of SO_REUSEPORT.

The socket option reuse port is available on Linux >3.9, and different from BSD it implements a simple load balancer. This obsoleted my thread balancer but it still useful on !Linux. This option is also nicer since it works for process as well.

With 80 cores there’s still the chance that the OS scheduler put most of your threads on the same cores, and maybe even move them when under load. So an option for setting a CPU affinity was also added, this allows for each work be pinned to one or more cores evenly. It uses the same logic as uwsgi.

Now that WSGI module supported all these features preview 3 of benchmarks came out and the results where still terrible… further investigation revealed that a variable supposed to be set with CPU core count was set to 8 instead of 80. I’m sure all this work did improve performance for servers with a lots of cores so in the end the wrong interpretation was good after all

Preview 4 came out and we are back to the top, I’ll do another post once it’s final.

Code name “to infinity and beyond” came to head due scalability options it got

Last but not least I did my best to get rid of doxygen missing documentation warnings.

Have fun https://github.com/cutelyst/cutelyst/archive/v1.6.0.tar.gz


Categories: FLOSS Project Planets

Jeff Geerling's Blog: Composer BoF at DrupalCon Baltimore

Planet Drupal - Tue, 2017-04-25 16:07

Tomorrow (Wednesday, April 25), I'm leading a Birds of a Feather (BoF) at DrupalCon Baltimore titled Managing Drupal sites with Composer (3:45 - 4:45 p.m. in room 305).

I've built four Drupal 8 websites now, and for each site, I have battle scars from working with Composer (read my Tips for Managing Drupal 8 projects with Composer). Even some of the tools that I use alongside composer—for project scaffolding, managing dependencies, patching things, etc.—have changed quite a bit over the past year.

As more and more Drupal developers adopt a Composer workflow for Drupal, we are solving some of the most painful problems.

For example:

Categories: FLOSS Project Planets

Valuebound: How to add Custom JS / CSS to Drupal 7 page in theme for a better user experience

Planet Drupal - Tue, 2017-04-25 14:25

Adding JS adds dynamic presentation effects to a theme for a better user experience.

In One of the project when i had been asked to display a custom error message to the User login page, when user tried with 3 or more failed attempt. On the other hand we can handle the error using form_set_error or drupal_set_message. What if we want to include that error message to be displayed on specific position.there comes add_js into picture. I have created a hook_form_alter inside that called a custom form validation. Under that custom form validation by checking the user login count, calling an js to the user page. Sleek and simple way to do the JS

Categories: FLOSS Project Planets

DataCamp: Keras Cheat Sheet: Neural Networks in Python

Planet Python - Tue, 2017-04-25 14:10

Keras is an easy-to-use and powerful library for Theano and TensorFlow that provides a high-level neural networks API to develop and evaluate deep learning models.

We recently launched one of the first online interactive deep learning course using Keras 2.0, called "Deep Learning in Python.

Now, DataCamp has created a Keras cheat sheet for those who have already taken the course and that still want a handy one-page reference or for those who need an extra push to get started.

In short, you'll see that this cheat sheet not only presents you with the six steps that you can go through to make neural networks in Python with the Keras library.

In no time, this Keras cheat sheet will make you familiar with how you can load datasets from the library itself, preprocess the data, build up a model architecture, and compile, train, and evaluate it. As there is a considerable amount of freedom in how you build up your models, you'll see that the cheat sheet uses some of the simple key code examples of the Keras library that you need to know to get started with building your own neural networks in Python.

Furthermore, you'll also see some examples of how to inspect your model, and how you can save and reload it. Lastly, you’ll also find examples of how you can predict values for test data and how you can fine tune your models by adjusting the optimization parameters and early stopping.

Everything you need to make your first neural networks in Python with Keras!

Also, don't miss out on our scikit-learn cheat sheet, NumPy cheat sheet and Pandas cheat sheet!

Categories: FLOSS Project Planets

aleksip.net: Does Drupal have a minor upgrade problem?

Planet Drupal - Tue, 2017-04-25 13:47
Drupal 8 has a new upgrade model, and the promise is to make upgrades easy forever. The idea behind the upgrade model is great, and has already been proven in other projects like Symfony. However, there might still be some issues that need to be solved, as demonstrated by the recent 8.3 release and the security release that followed it.
Categories: FLOSS Project Planets
Syndicate content