Feeds

How To Code RNN And LSTM Neural Networks In Python In [1]: import matplotlib.pyplot as plt import numpy as np import pandas as pd import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers tf.__version__ Out[1]: '2.3.1'

Check out following links if you want to learn more about Pandas and Numpy.

Pandas

Numpy Basics

What's so special about text?

Text is categorized as Sequential data: a document is a sequence of sentences, each sentence is a sequence of words, and each word is a sequence of characters. What is so special about text is that the next word in a sentence depends on:

1. Context: which can extend long distances before and after the word, aka long term dependancy.
2. Intent: different words can fit in the same contexts depending on the author's intent.

What do we need?

We need a neural network that models sequences. Specifically, given a sequence of words, we want to model the next word, then the next word, then the next word, ... and so on. That could be on a sentence, word, or character level. Our goal can be to just make a model to predict/generate the next word, like in unsupervised word embeddings. Alternatively, we could just map patterns in the text to associated labels, like in text classifications. In this notebook, we will be focusing on the latter. However, the networks used for either are pretty similar. The role of the network is most important in processing the textual input, extracting, and modelling the linguistic features. What we then do with these features is another story.

Recurrent Neural Networks (RNNs)

A Recurrent Neural Network (RNN) has a temporal dimension. In other words, the prediction of the first run of the network is fed as an input to the network in the next run. This beautifully reflects the nature of textual sequences: starting with the word "I" the network would expect to see "am", or "went", "go" ...etc. But then when we observe the next word, which let us say, is "am", the network tries to predict what comes after "I am", and so on. So yeah, it is a generative model!

Reber Grammar Classification

Let's start by a simple grammar classification. We assume there is a linguistic rule that characters are generated according to. This is a simple simulation of grammar in our natural language: you can say "I am" but not "I are". More onto Reber Grammar > here.

Defining the grammar

Consider the following Reber Grammar:

Reber Grammar

Let's represent it first in Python:

In [1]: default_reber_grammar=[ [("B",1)], #(state 0) =B=> (state 1) [("T", 2),("P", 3)], # (state 1) =T=> (state 2) or =P=> (state 3) [("X", 5), ("S", 2)], # (state 2) =X=> (state 5) or =S=> (state 2) [("T", 3), ("V", 4)], # (state 3) =T=> (state 3) or =V=> (state 4) [("V", 6), ("P", 5)], # (state 4) =V=> (state 6) or =P=> (state 5) [("X",3), ("S", 6)], # (state 5) =X=> (state 3) or =S=> (state 6) [("E", None)] # (state 6) =E=> <EOS> ]

Let's take this a step further, and use Embedded Reber Grammar, which simulates slightly more complicated linguistic rules, such as phrases!

In [2]: embedded_reber_grammar=[ [("B",1)], #(state 0) =B=> (state 1) [("T", 2),("P", 3)], # (state 1) =T=> (state 2) or =P=> (state 3) [(default_reber_grammar,4)], # (state 2) =REBER=> (state 4) [(default_reber_grammar,5)], # (state 3) =REBER=> (state 5) [("P", 6)], # (state 4) =P=> (state 6) [("T",6)], # (state 5) =T=> (state 3) [("E", None)] # (state 6) =E=> <EOS> ]

Now let's generate some data using these grammars:

Generating data In [3]: def generate_valid_string(grammar): state = 0 output = [] while state is not None: char, state = grammar[state][np.random.randint(len(grammar[state]))] if isinstance(char, list): # embedded reber char = generate_valid_string(char) output.append(char) return "".join(output) In [4]: def generate_corrupted_string(grammar, chars='BTSXPVE'): '''Substitute one character to violate the grammar''' good_string = generate_valid_string(grammar) idx = np.random.randint(len(good_string)) good_char = good_string[idx] bad_char = np.random.choice(sorted(set(chars)-set(good_char))) return good_string[:idx]+bad_char+good_string[idx+1:]

Let's define all the possible characters used in the grammar.

In [5]: chars='BTSXPVE' chars_dict = {a:i for i,a in enumerate(chars)} chars_dict Out[5]: {'B': 0, 'T': 1, 'S': 2, 'X': 3, 'P': 4, 'V': 5, 'E': 6}

One hot encoding is used to represent each character with a vector so that all vectors are equally far away from each other. For example,

In [6]: def str2onehot(string, num_steps=12, chars_dict=chars_dict): res = np.zeros((num_steps, len(chars_dict))) for i in range(min(len(string), num_steps)): c = string[i] res[i][chars_dict[c]] = 1 return res

Now let's generate a dataset of valid and corrupted strings

In [7]: def generate_data(data_size=10000, grammar=embedded_reber_grammar, num_steps=None): good = [generate_valid_string(grammar) for _ in range(data_size//2)] bad = [generate_corrupted_string(grammar) for _ in range(data_size//2)] all_strings = good+bad if num_steps is None: num_steps = max([len(s) for s in all_strings]) X = np.array([str2onehot(s) for s in all_strings]) l = np.array([len(s) for s in all_strings]) y = np.concatenate((np.ones(len(good)), np.zeros((len(bad))))).reshape(-1, 1) idx = np.random.permutation(data_size) return X[idx], l[idx], y[idx] In [9]: np.random.seed(42) X_train, seq_lens_train, y_train = generate_data(10000) X_val, seq_lens_val, y_val = generate_data(5000) X_train.shape, X_val.shape Out[9]: ((10000, 12, 7), (5000, 12, 7))

We have 10,000 words, each with 12 characters, and maximum of 7 unique letters (i.e. BTSXPVE)

Building the model

source

In [18]: x = layers.Input(shape=(12, 7)) # we define our input's shape # first we define our RNN cells to use in the RNN model # let's keep the model simple ... cell = layers.SimpleRNNCell(4, activation='tanh') # ... by just using 4 units (like 4 units in hidden layers) rnn = layers.RNN(cell) rnn_output = rnn(x)

We use tanh activation function to make the prediction between -1 and 1 the resulting activation between -1 and 1 is then weighted to finally give us the features to use in making our predictions

We finally add a fully connected layer to map our rnn outputs to the 0-1 classification output. We use a sigmoid function to map the prediction to probabilities between 0 and 1.

In [19]: output = layers.Dense(units=1, activation='sigmoid')(rnn_output) In [20]: # let's compile the model model = keras.Model(inputs=x, outputs=output) # loss is binary cropss entropy since this is a binary classification task # and evaluation metric as f1 model.compile(loss="binary_crossentropy", metrics=["accuracy"]) model.summary() Model: "functional_3" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_2 (InputLayer) [(None, 12, 7)] 0 _________________________________________________________________ rnn_1 (RNN) (None, 4) 48 _________________________________________________________________ dense_1 (Dense) (None, 1) 5 ================================================================= Total params: 53 Trainable params: 53 Non-trainable params: 0 _________________________________________________________________

We have 12 characters in each input, and 4 units per RNN cell, so we have a total of 12x4=48 parameters to learn + 5 more parameters from the fully connected (FC) layer.

In [21]: # we train the model for 100 epochs # verbose level 2 displays more info while trianing H = model.fit(X_train, y_train, epochs=100, verbose=2, validation_data=(X_val, y_val)) In [20]: def plot_results(H): results = pd.DataFrame({"Train Loss": H.history['loss'], "Validation Loss": H.history['val_loss'], "Train Accuracy": H.history['accuracy'], "Validation Accuracy": H.history['val_accuracy'] }) fig, ax = plt.subplots(nrows=2, figsize=(16, 9)) results[["Train Loss", "Validation Loss"]].plot(ax=ax[0]) results[["Train Accuracy", "Validation Accuracy"]].plot(ax=ax[1]) ax[0].set_xlabel("Epoch") ax[1].set_xlabel("Epoch") plt.show() In [38]: plot_results(H) LSTM

Long short-term memory employs logic gates to control multiple RNNs, each is trained for a specific task. LSTMs allow the model to memorize long-term dependancies and forget less likely predictions. For example, if the training data had "John saw Sarah" and "Sarah saw John", when the model is given "John saw", the word "saw" can predict "Sarah" and "John" as they have been seen just after "saw". LSTM allows the model to recognize that "John saw" is going to undermine the possibility for "John", so we won't get "John saw John". Also we won't get "John saw John saw John saw John ..." as the model can predict that what comes after the word after saw, is the end of the sentence.

source

Now we will apply bidirectional LSTM (that looks both backward and forward in the sentence) for text classification.

Sentiment Analysis: IMDB reviews

source

NEVER train two models on the same kernel session. We already trained the reber grammar one, so we need to restart the kernel first.

Loading the data In [2]: !pip install -q tensorflow_datasets In [3]: import tensorflow_datasets as tfds In [4]: dataset, info = tfds.load('imdb_reviews', with_info=True, as_supervised=True) train_dataset, test_dataset = dataset['train'], dataset['test'] Processing the data

Now that we have downloaded the data, we now can go ahead and:

1. (optional) take a small sample of the data, since this is just a demo!
2. Align the reviews with their labels
3. Shuffle the data

In [5]: train = train_dataset.take(4000) test = test_dataset.take(1000) In [6]: # to shuffle the data ... BUFFER_SIZE = 4000 # we will put all the data into this big buffer, and sample randomly from the buffer BATCH_SIZE = 128 # we will read 128 reviews at a time train = train.shuffle(BUFFER_SIZE).batch(BATCH_SIZE) test = test.batch(BATCH_SIZE)

prefetch: to allow the later elements to be prepared while the current elements are being processed.

In [7]: train = train.prefetch(BUFFER_SIZE) test = test.prefetch(BUFFER_SIZE) Text Encoding

Each word in the sentence is going to be replaced with its corresponding index in the vocabulary.

In [8]: VOCAB_SIZE=1000 # assuming our vocabulary is just 1000 words encoder = layers.experimental.preprocessing.TextVectorization(max_tokens=VOCAB_SIZE) encoder.adapt(train.map(lambda text, label: text)) # we just encode the text, not the labels In [9]: # here are the first 20 words in our 1000-word vocabulary vocab = np.array(encoder.get_vocabulary()) vocab[:20] Out[9]: array(['', '[UNK]', 'the', 'and', 'a', 'of', 'to', 'is', 'in', 'i', 'it', 'this', 'that', 'br', 'was', 'as', 'with', 'for', 'but', 'movie'], dtype='<U14') In [10]: example, label = list(train.take(1))[0] # that's one batch len(example) Out[10]: 128 In [11]: example[0].numpy() Out[11]: b'There have been so many many films based on the same theme. single cute girl needs handsome boy to impress ex, pays him and then (guess what?) she falls in love with him, there\'s a bit of fumbling followed by a row before everyone makes up before the happy ending......this has been done many times.<br /><br />The thing is I knew this before starting to watch. But, despite this, I was still looking forward to it. In the right hands, with a good cast and a bright script it can still be a pleasant way to pass a couple of hours.<br /><br />this was none of these.<br /><br />this was dire.<br /><br />A female lead lacking in charm or wit who totally failed to light even the slightest spark in me. I truly did not care if she "got her man" or remained single and unhappy.<br /><br />A male lead who, after a few of his endless words of wisdom, i wanted to kill. Just to remove that smug look. i had no idea that leading a life of a male whore was the path to all-seeing all-knowing enlightenment.<br /><br />A totally unrealistic film filled with unrealistic characters. none of them seemed to have jobs, all of them had more money than sense, a bridegroom who still goes ahead with his wedding after learning that his bride slept with his best friend....plus "i would miss you even if we had never met"!!!!! i could go on but i have just realised that i am wasting even more time on this dross.....I could rant about introducing a character just to have a very cheap laugh at the name "woody" but in truth that was the only remotely humorous thing that happened in the film.' In [12]: encoded_example = encoder(example[:1]).numpy() encoded_example Out[12]: array([[ 49, 26, 78, 36, 107, 107, 92, 417, 21, 2, 165, 810, 593, 988, 241, 795, 1, 429, 6, 1, 1, 1, 90, 3, 91, 495, 48, 56, 646, 8, 113, 16, 90, 222, 4, 197, 5, 1, 1, 33, 4, 1, 157, 336, 151, 57, 157, 2, 659, 1, 46, 78, 218, 107, 1, 13, 2, 144, 7, 9, 782, 11, 157, 1, 6, 104, 18, 475, 11, 9, 14, 122, 289, 971, 6, 10, 8, 2, 212, 946, 16, 4, 50, 185, 3, 4, 1, 227, 10, 69, 122, 28, 4, 1, 97, 6, 1, 4, 367, 5, 1, 13, 11, 14, 683, 5, 1, 13, 11, 14, 1, 13, 4, 634, 480, 1, 8, 1, 42, 1, 37, 432, 901, 6, 752, 55, 2, 1, 1, 8, 70, 9, 347, 118, 22, 425, 43, 56, 175, 40, 121, 42, 1, 593, 3, 1, 13, 4, 1, 480, 37, 101, 4, 178, 5, 23, 1, 609, 5, 1, 9, 449, 6, 485, 41, 6, 1, 12, 1, 158, 9, 63, 58, 326, 12, 813, 4, 115, 5, 4, 1, 1, 14, 2, 1, 6, 1, 1, 1, 13, 4, 432, 1, 20, 1, 16, 1, 103, 683, 5, 95, 463, 6, 26, 1, 32, 5, 95, 63, 51, 270, 71, 275, 4, 1, 37, 122, 278, 1, 16, 23, 1, 101, 1, 12, 23, 1, 1, 16, 23, 108, 1, 9, 60, 731, 25, 55, 43, 73, 63, 114, 1, 9, 96, 131, 21, 18, 9, 26, 41, 1, 12, 9, 214, 1, 55, 51, 59, 21, 11, 1, 96, 1, 45, 1, 4, 109, 41, 6, 26, 4, 52, 831, 500, 31, 2, 391, 1, 18, 8, 883, 12, 14, 2, 64, 1, 1, 144, 12, 571, 8, 2, 20]]) Creating the model

In [13]: model = tf.keras.Sequential([ encoder, # the encoder tf.keras.layers.Embedding( input_dim=len(encoder.get_vocabulary()), output_dim=64, # Use masking to handle the variable sequence lengths mask_zero=True), tf.keras.layers.Bidirectional(layers.LSTM(64)), # making LSTM bidirectional tf.keras.layers.Dense(32, activation='relu'), # FC layer for the classification part tf.keras.layers.Dense(1) # final FC layer ])

Let's try it out!

In [14]: sample_text = ('The movie was cool. The animation and the graphics ' 'were out of this world. I would recommend this movie.') predictions = model.predict(np.array([sample_text])) print(predictions[0]) [-0.00052149]

yeah yeah, we haven't trained the model yet.

Compiling & training the model In [15]: # we will use binary cross entropy again because this is a binary classification task (positive or negative) # we also did not apply a sigmoid activation function at the last FC layer, so we specify that the # are calculating the cross entropy from logits model.compile( loss=tf.keras.losses.BinaryCrossentropy(from_logits=True), # adam optimizer is more efficient (not always the most accurate though) optimizer=tf.keras.optimizers.Adam(1e-4), metrics=['accuracy'] ) In [16]: model.summary() Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= text_vectorization (TextVect (None, None) 0 _________________________________________________________________ embedding (Embedding) (None, None, 64) 64000 _________________________________________________________________ bidirectional (Bidirectional (None, 128) 66048 _________________________________________________________________ dense (Dense) (None, 32) 4128 _________________________________________________________________ dense_1 (Dense) (None, 1) 33 ================================================================= Total params: 134,209 Trainable params: 134,209 Non-trainable params: 0 _________________________________________________________________

Wow that's a lot of parameters!

In [17]: H2 = model.fit(train, epochs=25, validation_data=test) In [21]: plot_results(H2)

It works! We stopped after only 25 epochs, but obviously still has plenty of room for fitting with more epochs.

Summary & Comments

1. Text is a simply a sequential data.
2. RNN-like models feed the prediction of the current run as input to the next run.
3. LSTM uses 4 RNNs to handel more complex features of text (e.g. long-term dependancy)
4. Bidirectional models can provide remarkably outperform unidirectional models.
5. You can stack as many LSTM layers as you want. It is just a new LEGO piece to use when building your NN :)

A new website containing introductory videos and slide decks is now available for your perusal at ess-intro.github.io. It provides a series of introductions to the excellent Emacs Speaks Statistics (ESS) mode for the Emacs editor.

This effort started following my little tips, tricks, tools and toys series of short videos and slide decks “for the command-line and R, broadly-speaking”. Which I had mentioned to friends curious about Emacs, and on the ess-help mailing list. And lo and behold, over the fall and winter sixteen of us came together in one GitHub org and are now proud to present the initial batch of videos about first steps, installing, using with spaceemacs, customizing, and org-mode with ESS. More may hopefully fellow, the group is open and you too can join: see the main repo and its wiki.

This is in fact the initial announcement post, so it is flattering that we have already received over 350 views, four comments and twenty-one likes.

We hope it proves to be a useful starting point for some of you. The Emacs editor is quite uniquely powerful, and coupled with ESS makes for a rather nice environment for programming with data, or analysing, visualising, exploring, … data. But we are not zealots: there are many editors and environments under the sun, and most people are perfectly happy with their choice, which is wonderful. We also like ours, and sometimes someone asks ‘tell me more’ or ‘how do I start’. We hope this series satisifies this initial curiousity and takes it from here.

With that, my thanks to Frédéric, Alex, Tyler and Greg for the initial batch, and for everybody else in the org who chipped in with comments and suggestion. We hope it grows from here, so happy Emacsing with R from us!

There is a serious problem with Dreamwidth, which is impeding access for many RSS reader tools.

This started at around 0500 UTC on Wednesday morning, according to my own RSS reader cron job. A friend found #43443 in the DW ticket tracker, where a user of a minority web browser found they were blocked.

Local tests demonstrated that Dreamwidth had applied blocking by the HTTP User-Agent header, and were rejecting all user-agents not specifically permitted. Today, this rule has been relaxed and unknown user-agents are permitted. But user-agents for general http client libraries are still blocked.

I'm aware of three unresolved tickets about this: #43444 #43445 #43447

We're told there by a volunteer member of Dreamwidth's support staff that this has been done deliberately for "blocking automated traffic". I'm sure the volunteer is just relaying what they've been told by whoever is struggling to deal with what I suppose is probably a spam problem. But it's still rather unsatisfactory.

I have suggested in my own ticket that a good solution might be to apply the new block only to posting and commenting (eg, maybe, by applying it only to HTTP POST requests). If the problem is indeed spam then that ought to be good enough, and would still let RSS readers work properly.

I'm told that this new blocking has been done by "implementing" (actually, configuring or enabling) "some AWS rules for blocking automated traffic". I don't know what facilities AWS provides. This kind of helplessness is of course precisely the kind of thing that the Free Software movement is against and precisely the kind of thing that proprietary services like AWS produce.

I don't know if this blog entry will appear on planet.debian.org and on other people's reader's and aggregators. I think it will at least be seen by other Dreamwidth users. I thought I would post here in the hope that other Dreamwidth users might be able to help get this fixed. At the very least other Dreamwidth blog owners need to know that many of their readers may not be seeing their posts at all.

If this problem is not fixed I will have to move my blog. One of the main points of having a blog is publishing it via RSS. RSS readers are of course based on general http client libraries and many if not most RSS readers have not bothered to customise their user-agent. Those are currently blocked.

comments

Last year I decided to add an improvement to my home office, a treadmill. I thought about during the total lock-down we suffered in Spain in spring 2020. The reason for not trying it earlier was that I was skeptical about keeping my productivity level while walking on the treadmill. Finally I made the decision to try out and asked for one to The Three Wise Men. They were kind enough to bring it.

After a few weeks using it, I would like to report about it since several of my colleagues at MBition/Daimler asked me about it. I hope that reading about my experience is useful to you.

Select the right treadmill

I started by doing some research on which treadmill to buy. The main characteristics that led my decision were:

• Walking vs running: I wanted a treadmill for walking, not for running. There are quite a few models that has 6 to 7 km/h as speed limit and no support or add-ons to hold on to while walking. The intention is to avoid any clashing with my standing desk converter.
• I am a big man so I need a big treadmill: the market offers treadmills with different length, surface and weight limits. I searched for one that can afford a big man like me and that was as wide as as possible. Falling from it was a concern.
• Remote control: since you will be walking while working, you need a way to control the treadmill remotely, either with a remote control, an app or similar. I selected one with a remote control.
• With wheels: I need to move the treadmill often since I alternate using the treadmill with working standing up or seated. I wanted one with wheels so it is easier to move. Treadmills are heavy.
• Limited investment: my idea was to try out first so I did not want to invest a lot. If it works, I will invest more in a replacement when needed.
• Pay attention to the noise level: my treadmill is noisy although not enough to be annoying or to be heard by others while in video calls. Check the specifications, I did not so I simply was a little lucky..

I bought this model, produced by 2WD, ASIN/model B08GLTX8LK, through Amazon for €299. I had no issues with the purchase.

Usage

I use the treadmill between 3 to 4 hours a day. I started using it between 5 and 6 hours but soon I got hurt in one foot which forced me to stop for some time. First lesson learnt, take it easy, specially if, like myself, you are not in great shape.

At first I ended up so tired that it was hard for me to keep good levels of concentration during the last part of any working day. As I got more used to using the treadmill, I felt more energetic in general so that tiredness is little by little going away. My productivity is increasing overtime compared to the first few days. The last few weeks I even have energy left to do some soft exercise after work.

I started walking at 2 km/h. I increased to 2.5 km/h after 2 weeks. A few weeks later I tried 3 km/h but it was harder for me to concentrate, it is too fast for now. There is a mismatch by the way between the speed and the distance walked marked by the treadmill. I think that the average speed I walk is a little below 2.5 km/h. In am current walking between 3 and 4 hours a day which is around 7-8.5 kms (10k steps is around 8 kms). In any case, I try to reach the 6.5 kms mark every day. I am accomplishing it regularly during the last few weeks.

Can I work normally while walking?

This is the question most colleagues ask me. The answer is yes… for most activities.

My experience is that I can work normally while walking, keeping good productivity levels, while performing most type of activities, specially during most meetings.

In which ones I have seen a reduction of productivity?

I detected a productivity decrease in those activities which require high levels of concentration or creativity like complex meetings with many people that I facilitate, those times when I need to come up with new ideas, analysis of complex data, etc. The good news is that I did detect this productivity reduction early. The longer I have been using the treadmill though, the more type tasks I can do without noticing a productivity reduction. So I plan to try again during the coming weeks some of those activities that I dropped early on.

Walking is helping me a lot to go through those days with back to back meetings. I pay better attention and get less bored compared to standing up. As I get more used to walking, my energy levels during the afternoons are increasing, as mentioned, which helps me to get through the last couple of hours during long working days. It was not like that at first though, so plan accordingly.

The unexpected surprises

The treadmill uses IR to connect to the remote control which interferes with an IR based device I had at my office, the HDMI hub. I use two laptops and two monitors and used the HDMI hub to have one of the two screens assigned to one of the laptops while the default config is to have both monitors assigned to my working laptop. I did not find a way to prevent such interference so I had to remove the HDMI hub. I am looking for alternatives although the impact in my daily workflow is low. The problems were easy to detect because every time I turned the treadmill on, one of my screens turned off although both laptops detected it.. So the takeaway is: watch out for IR interference.

I trickier surprise for which I have not find a solution is the problems associated with the PLC. I have my office connected to the living room, where the fiber router is, through PLC. This has been my default set up since years in several houses. Something related with the treadmill makes the link between both PLC devices unstable which ends up in internet issues. I have a backup for cable connectivity which is a wifi replicator and a specific wifi network for the devices I use for work (in other jobs I used to play with boards which required a testing network. So sometimes I have to use wifi instead of cable because the link between both PLC devices get unstable. It took me a while to detect the source of issues. After testing all the current current plugs at my office, I ended up finding one which makes the connectivity issues tolerable. Still I am searching for a solution here..

The takeaway is that the treadmill consumes quite some power and might have spurious looses that might turn network connectivity through PLC unstable. This is not the first time I suffer from this. The first mobile heat device I had at my office provoke the same effect.

Other points to consider

Some days I start walking and some others I walk after lunch on when I have back to back meetings. I tend to change the set up putting or removing the treadmill no more than once a day. Usually I have an additional set up change when I push down my Varidesk to work seated.

The standing desk converter I have is big enough to allow me to hold on to it when I do not have my hands on the keyboard. That have saved me from falling down during meetings more than once. Be careful, falling down is a real risk. Take precautions.

Get professional advice about walking with or without shoes. In my case, I walk without shoes. This decision might change in the future. My knees are awful due to injuries during my basketball times. Pay attention to the way you walk. When you get tired, your walking style changes and in might end up hurting.

My treadmill has a limit of 100 minutes. When it reaches that threshold, it stops. I was annoyed at first but it has ended up being a great thing. I used to work in cycles of 110 or 120 minutes but sometimes I loose track of time and this hard limit helps. While walking, my working cycles are shorter. 100 min which helps my concentration specially at the end of the day.

Obviously during meetings participants notice that I am walking. Be ready for questions and jokes. Having a treadmill is not mainstream yet. Ah, and do not fall while in a meeting.

Conclusions

At first I overdid it, got hurt and tired fast so my productivity in the afternoons dropped. In my second try I took it easier and the experience is getting better overtime. Six weeks after being back at using the treadmill again I am feeling better. I will soon be ready to increase the time walking and the type of activities I execute while on the treadmill.

I think it will take me another few weeks before the benefits for my health and shape become evident. I can see early signs though. So in general I find the experience positive so far although I expect to get better after a few more weeks when, in addition to the walking done in the treadmill, I have energy to complement such exercise with additional one on regular basis instead of just some days. The same applies to the level of concentration while doing the most demanding tasks.

The connectivity issues I am experiencing are unfortunate although tolerable. I need to find a solution.

So yes, I totally recommend to get yourself a treadmill. Results will show up sooner or later depending on your prior shape. I think that if you are in better shape than I was when I got it, which is not hard, results will be noticeable in a couple of months, probably less.

Featuring a video from Acro Media’s YouTube tutorial series Tech Talk, this article will walk you through setting up an awesome Drupal Commerce product catalogue using Search API and Solr, and then adding various ways of filtering the results (product search, sorting options and Facet categories).

Ten steps to creating a product catalogue with Search API, Solr & Facets

These 10 steps will get you a Search API index using Solr, setup a View to display the results of the index, and then implemented 3 different ways to filter the results (search, sorting and Facets).

• Things you need: a running Solr server, Drupal 8 (or higher) with Commerce, Search API. Solr and Facets modules installed.
• Install/start Solr and enable the Commerce, Search API, Solr and Facets modules.
• Setup a basic Commerce store.
• Add a Search API server and index.
• Create a View for the catalogue.
• Add Fulltext datasource fields for a catalogue search field.
• Set up the catalogue search field within the catalogue View.
• Add more datasource fields for sorting options.
• Add Facets to the catalogue.
• Configure Facets to maintain active facets when using filters.

The end result of this guide will be a catalogue that functions in the same way as our Urban Hipster Drupal Commerce demo site’s catalogue. You can try it here. If you don’t already know why we use Search API, Solr and Facets for catalogues, check out this article to get up to speed.

Even though we’re going to be using products, once you understand how it works you’ll be able to apply the same method for other type of content such as a blog, videos, resources, and more. The datasource can change but the process is the same.

Let's get started! Follow along with this video or skip below for a written guide.

What you need before starting

1. A running Solr server (Solr 6.4+)
   This tutorial assumes you have Solr running and can make a new core.
   
   
2. Drupal 8 installed with the following modules:
   
   TIP: Get most of what you need quickly with Commerce Kickstart. Note that you will still need to install the Facets module after getting your Kickstart package.
   
   
   • Commerce
     composer require drupal/commerce
   • Search API
     composer require drupal/search_api
   • Solr
     NOTE: This module requires you're running Solr 6.4+ and PHP 7
     composer require drupal/search_api_solr
   • Facets
     composer require drupal/facets

Getting started

1. Install/start Solr and add a new core that we’ll use later.
   
   
2. Enable the Commerce, Search API, Solr and Facets modules.

Setup a basic Commerce store For this tutorial, get your site and basic store set up by doing the following:

1. Add a product category taxonomy vocabulary that is at least 2 levels deep.
   If we use clothing as an example, we might have Men and Women as the first level, then t-shirts, shorts and shoes as the second level for each.
   
   
2. Setup your basic Commerce store, product types and product variation types.
   If you’re new to Commerce, take a look at their documentation to get up and running quickly.
   
   NOTE: Make sure to add the taxonomy vocabulary you created as a ‘taxonomy reference’ field to your Product Type.
   
   
3. Add 10 or more simple products for testing the catalogue as we go.

Add a Search API server and index Search API server

Admin: Configuration > Search and metadata > Search API
Admin menu path: /admin/config/search/search-api

1. Click ‘Add server’.
   
   
2. Configure the server.
   1. Name your server and enable it.
   2. Set ‘Solr’ as the server ‘Backend’.
   3. Configure the Solr connector.
      The defaults are usually fine. The main things to add are:
      • Solr connector = ‘Standard’.
      • Solr core = Whatever you named your core.
   4. Under ‘Advanced’, check ‘Retrieve result data from Solr’.
   5. Look over the remaining settings and update if you need to.

Search API index

Admin:Configuration > Search and metadata > Search API
Admin menu path:/admin/config/search/search-api

The index is where you set what data source is used by Search API. Eventually, you’ll also specify specific fields to be used for filtering the displayed results.

1. Click ‘Add index’.
   
   
2. Configure the index.
   1. Name your index.
   2. Data source should be ‘Product’
      This can be anything, but we’re creating a Commerce catalogue and so we want to use the store products.
   3. Select the server you just created.
   4. Save. Don’t add any fields for now, we’ll do that later.
   5. Go to the ‘View’ tab and index your results. This will index all of the products you have added so far.

Create a View for the catalogue

Admin:Structure > Views
Admin menu path:/admin/structure/views

The View will use the data source we’ve identified in our index and allow us to create a catalogue with it, and then assign ways of filtering the catalogue results (i.e. a search field and/or facets).

1. Create a new View.
   1. View Settings, select your index.
   2. Create a page (this will become our catalog).
2. View Display settings.
   1. Format > Show
      Set as ‘Rendered entity’, then in the settings, set your product types to use a ‘Teaser’ view mode instead of the default.
      
      NOTE: If you don't see the 'Rendered entity' option, this seems to be a bug with Views and happens when you don't select your index before checking the 'Create a page' checkbox. To fix this, just refresh your page to start over. If that doesn't work, flush your caches.
      
      NOTE:You may need to create this view mode if it doesn’t already exist.
      
      NOTE:You could alternately use Fields instead of view modes, but I like to keep my product display settings all within the product type’s display settings. Then you can potentially customize the display per product type if you have more than one.
3. Save the view .
   These basic settings should give us our overall catalog. You can confirm by previewing the view or visiting the page you just created.

Add Fulltext datasource fields for a catalog search field

Now we’ll start setting up a Fulltext search field to let our users filter results using a product search field. The first thing we need to do is add some datasource fields to our index that the search will use.

1. Go to your Search API Index and go to the Fields tab.
   
   
2. Add Fulltext fields that you would like to be searchable (such as Title, SKU, Category taxonomy name, etc.).
   Here’s an example for adding the title:
   1. Click ‘Add fields’.
   2. Under the ‘Product’ heading, click ‘Add’ beside the ‘Title’ field.
      
      NOTE: If you’re adding a different field instead, you may need to drill down further into the field by clicking ( + ) next to the field. For example, to make a taxonomy term a searchable field, you would go to Your Vocabulary > Taxonomy Term > Name .
      
      
   3. Click ‘Done’.
   4. Set the field ‘Type’ to ‘Fulltext’.
      This is an important step as only Fulltext fields are searchable with the user-entered text search we are currently setting up.
      
      NOTE: Under the fields section is a ‘Data Types’ section. You can open that to read information about each available type.
      
      
   5. Optionally change the ‘Boost’ setting.
      If you have more than one Fulltext field, the boost setting allows you to give a higher priority to specific fields for when the terms are being searched.
      
      For example, multiple products could have a similar title, but each product would have an individual SKU. In this case, SKU could be given a higher boost than title to make sure that search results based on the SKU come back first.
3. Next, add another field for the ‘Published’ status.
   
   
4. Once you’ve added this field, set it’s type as ‘Boolean’.
   
   
5. Reindex your data (from within the index view tab).

Set up the catalogue search field within the catalogue View

We can now set up the actual search field that our customers will use to find products, and use the datasource fields we added to our index to do this.

1. Go to your catalog View.
   
   
2. Under ‘Filter criteria’.
   1. Add ‘Fulltext search’ and configure its settings.
      • Check ‘Expose this filter to visitors, to allow them to change it’.
        IMPORTANT:This is what gives the user the ability to use this search field.
      • ‘Filter type to expose’, set as ‘Single filter’.
      • ‘Operator’, set as ‘Contains any of these words’.
      • ‘Filter identifier’, optionally adds an identifier into the url to identify a search term filter.
        (i.e. yoursite.com/products?your-filter-identifier=search-term)
      • Apply/save your settings.
   2. Add ‘Published’ and configure it so that it is equal to true.
      This uses the field we added to the index earlier to make sure the product is actually published. Chances are you don’t want unpublished results shown to your customers.
3. Under ‘Sort criteria’.
   1. Add ‘Relevance’.
   2. Configure so that the order is sorted ascending.
      This will show the more relevant results first (factoring in the boost you may have applied to your index fields).
4. Now we need to expose the search field to our customers. To do this:
   1. Open the ‘Advanced’ section of your catalog view.
   2. In the ‘Exposed Form’ area.
      • Set ‘Exposed form in block’ to ‘Yes’.
        This creates a block containing a search field that we can place on the site somewhere.
      • Set ‘Exposed form style’ to ‘Basic’ and update the settings. For now, the settings you might change are customizing the submit button text and maybe including a reset button.
5. Add the search block to your site.
   Admin menu path:/admin/structure/block
   
   
1. In your preferred region, click the ‘Place block’ button.
2. Find the Views block that starts with ‘Exposed form’ and click ‘Place block’.
   Its full name will be determined by you view’s machine name and page display name (i.e. Exposed form: products-page_1).
3. Configure the block as you see fit, and save.
6. Test your search!
   You should now be able to see the search field on your site frontend and try it out.

Add more datasource fields for sorting options

We can optionally sort the catalogue and search results with some additional sorting filters, such as sorting by Title, Price, Date added, etc. Let’s add the ability to sort our products by title with the option to choose ascending or descending order.

1. Go to your Search API Index fields and add another 'Title' field the same as you did earlier. However, this time you want to change the field ‘Type’ to ‘String’. You should now have two Title fields added, one as ‘Fulltext’ and one as ‘String’.
   
   NOTE: The field type can be different depending on what field you’re adding. If you’re adding a sorting field such as Price > Number, you might use the ‘Decimal’ field type.
   
   TIP: I would recommend changing the label for the new Title field to something like ‘Title for sorting’ so that it’s easier to identify later. You could even change the fulltext Title label to ‘Title for search’, just to keep them organized and easy to understand.
   
   
2. Reindex your data (from within the index view tab).
   
   
3. Go to your catalogue View.
   1. Under ‘Sort criteria’.
      • Add the new title datasource and configure it.
        • Check ‘Expose this sort to visitors, to allow them to change it’.
          IMPORTANT: This is what gives the user the ability to use this sorting option.
        • Add a label for the filter
        • Set the initial sorting method.
      • Add any additional sorting fields if you added more.
   2. Open the settings for the ‘Exposed form style’ (within the view’s ‘Advanced’ section).
      • Check ‘Allow people to choose the sort order’.
      • Update the labels as you see fit.
   3. Save your view!
      Refresh your catalogue page and you should now see sorting options available in the search block that you added earlier.
      
      TIP: If you DO NOT see the sorting options, this is a bug and is easily fixed. All you need to do is remove the search block and then re-add it.
      
      TIP: You can place this block in multiple regions of your site and hide the elements you don’t want to see with CSS. This way you can have a block with the site search and no filters in your header, and then also have another block on your catalog pages that shows the sorting filters but no search field.

Add Facets to the catalogue

The filters we added earlier can only be used one at a time, however, often we want to filter the results based on a number of different options. For example, if I’m browsing an online store looking for shoes of a certain style and size, I don’t want to see everything else the store has to offer. I want to be able to go to a ‘shoe’ category, then pick the ‘type’ of shoe that I’m after, and finally pick the ‘size’ of shoe that’s relevant to me. I want to see all of the results that fit that criteria. Facets lets you use taxonomy (and other datasources) to achieve this.

Let’s add a Facet that uses the taxonomy vocabulary we created in the initial store setup. This will be our main catalogue menu for narrowing down the product results. Each facet that is created creates a block that we can add into any region of our template.

1. Add a Search API index field for your taxonomy vocabulary. Set the field ‘Type’ as ‘String’.
   
   TIP: Like we did earlier, I would recommend renaming the label for this field to something like ‘Categories for Facet’.
   
   
2. Reindex your data (from within the index view tab).
   
   
3. Go to the Facets page.
   Admin: Configuration > Search and metadata > Facets
   Admin menu path: /admin/config/search/facets
   
   You should see a ‘Facet source’ available to use. When we created a View using our index, this is what added the Facet source here. Now that we have a source, we can create Facets to filter it.
   
   
4. Click ‘Add facet’.
   1. Choose the ‘Facet source’ to use.
   2. Select the index ‘Field’ that this Facet will use (i.e. Categories for Facet, or whatever you labelled your field).
   3. Name your Facet (i.e. Categories).
5. Configure the Facet.
   This will cover the basic settings that you will need. Start with this and then you can always play around with other settings later. Each setting has a pretty good description to help you understand what it does.
   1. Widget.
      Choose a ‘Widget’ for displaying the categories. For categories, I like to use ‘List of checkboxes’.
   2. Facet Settings.
      Check the following:
      • Transform entity ID to label.
      • Hide facet when facet source is not rendered.
      • URL alias as ‘cat’ (or whatever you like).
      • Empty facet behavior as ‘Do not display facet’.
      • Operator as ‘OR’.
      • Use hierarchy.
      • Enable parent when child gets disabled.
      • NOTE: the Facets Pretty Paths module can be used to give you nicer looking URL paths.
   3. Facet Sorting.
      Configure as you see fit. In this example, I would only check the following.These settings make sure that the taxonomy follows the same order that you have set within the vocabulary itself.
      • Sort by taxonomy term weight.
      • Sort order as ‘Ascending’.
   4. Save.
6. Add the Facet block to your site.
   Admin: Structure > Block layout
   Admin menu path: /admin/structure/block
   
   
   1. In your preferred region, click the ‘Place block’ button.
   2. Find the ‘Categories’ facet block (or whatever you named it) and click ‘Place block’.
   3. Configure the block as you see fit.
   4. Save.
7. Test your Facet!
   You should now see your Facet on the catalog page. Click the checkboxes and test out how it works!

One last thing...

The sites we've been building use the Facets Pretty Paths module for making nicer looking URLs with our catalog and filters. For a while we were plagued with a problem where, when the user selects a Facet category and then uses the sorting options, the Facets would uncheck and reset. This is obviously not good because the user is trying to sort the filtered down items, not the overall catalog. We need to be able to maintain the active facets when using the filters.

Luckily, a coworker came up with this nice little solutions that you can apply to your theme's .theme file. You just need to replace YOUR_THEME,YOUR-VIEW (i.e. products-page-1), and YOUR-PATH (i.e. products) in the code below. Ideally, this will be fixed within the module itself soon, but this will work while we wait.

/**
* Implements hook_form_alter().
*/
function YOUR_THEME_form_alter(&$form, FormStateInterface $form_state, $form_id) {
// Store - Product Listing view exposed form.
if ($form['#id'] == 'views-exposed-form-YOUR-VIEW') {
$current_path = \Drupal::request()->getRequestUri();

// If current path is within your catalog, correct the form action path.
if ((strpos($current_path, '/YOUR-PATH') === 0)) {
// Fix for views using facets with pretty paths enabled.
// Replace form action with current path to maintain active facets.
$form['#action'] = $current_path;
}
}
}

Done!

There you have it! You have now created a Search API index using Solr, setup a View to display the results of the index, and then implemented 3 different ways to filter the results (search, sorting and Facets). This is the start of an awesome product catalogue and you can expand on it with more datasource fields however you want. Cool!

Editor’s note: This article was originally published on July 12, 2018, and has been updated for freshness, accuracy and comprehensiveness.

 

For those developers, whose daily work benefits from the Django framework.

As you probably know, the Django framework is built and maintained by community members. This might give you the impression that Django is effortlessly built by itself, but the truth is that the organization needs strong management and efficient communication. Making major Django releases every 8 months and bug fixes every month is no small task! That’s why the Django Fellowship program exists and why Django Software Foundation needs support from the community – both technical and financial.

Moreover, the Django Software Foundation supports community development and outreach projects such as Django Girls.

You can read more about the DSF activities here.

Support Django!

JetBrains is participating in a Django fundraising campaign – this year is the fifth iteration of this initiative. Over the past four years, JetBrains PyCharm has raised more than $140,000 for the Django Software Foundation.

How does it work?

During this campaign, you can buy a new individual license for PyCharm Professional for 30% off, and the full purchase price will go to the DSF’s general fundraising efforts and the Django Fellowship program.

This campaign will help the DSF maintain a healthy Django project and continue contributing to their various outreach programs.

Only with our support can Django make sure that the Web framework you base your work on can grow to be even better in the coming years. Don’t hesitate to share the link to the fundraising campaign with your fellow developers, who benefit from Django. Together we can support Django a lot!

Support Django!

Stay at the edge of your efficiency with Django and PyCharm

PyCharm always tries to support the latest features in Django and other technologies and make working with them easier, so that you can spend more time and energy solving the really interesting problems in your projects.

To help you get even more out of PyCharm and Django, we often prepare tutorials and arrange webinars on Django-related topics. Here are some useful resources we have for learning about working with Django projects:

• Tutorial by Mukul Mantosh “Developing Django Application using AWS”
• “Django with PyCharm Tips and Tricks” playlist
• Webinar “Database Performance Tips with Django”
• Webinar “How to Build Real Time Interactions in Your Django 3 App”

The body field is dead jnettik Thu, 04/15/2021 - 09:56 Authoring Experience Drupal

Content authoring on the web has evolved. Compelling web content uses a variety of multimedia elements to engage users, tell stories, build brands, and share new ideas. Images, video clips, slideshows, static or parallax backgrounds, block quotes and text pullouts — more than ever, content creators need a tool that embraces an evolving medium and keeps pace with the author’s creativity. We believe Layout Paragraphs is that tool. But first, let’s all agree that the body field is dead.

What is the body field

Web editors and content authors who have been around over the last decade of digital media are intimately familiar with the body field. Many popular content management systems (looking at you, Drupal and Wordpress) feature a standard “post” or “content” type — like page, article, or blog post — right out of the box. These content types often have a couple of fields to fill out before you can publish your post, things like title, tags, friendly URL / slug, and of course body.

The body field, traditionally, is where all your content goes. In the early days of blogging and web publishing that was likely to be all or mostly text, but with time came images, videos, and eventually an array of other multimedia elements. And the body field adapted. Tokens became standard for plenty of web editors — cryptic chunks of text like [[nid:376 align:right]] that would be auto-magically replaced with other elements once you click Publish. WYSIWYG editors (What You See Is What You Get) started shipping with Insert image and Insert video buttons, and began including a variety of tools for encapsulating, aligning, and positioning lengths of text or various media elements. And while these accommodations started to connect content creators with the boundless possibilities of multimedia authoring, they were (and are) clumsy and unpredictable.

Structured content: Reduce, reuse, recycle

One major problem with all of these innovations is that once you click Publish, all that complex content still ends up in the body field — that is, saved into your database as one giant clump of complicated text, tokens, style codes, etc. If you’d like to publish another page with a similar look & feel, get ready to re-inject all of your tokens or go through the same WYSIWYG click-a-thon to reestablish your styles and layout. Thinking of producing a list of all the images used in your posts? Good luck! With all content mushed together into a single field, your site doesn’t “know” the difference between a paragraph of text, an image, or a video.

A structured approach to content organizes each individual element — a pane of text, an image, slideshow, video, or a “donate now” banner — as its own self-contained entity which, ideally, could then be placed within flexibly configured regions on a page. Creating other, similar pages then becomes as simple as swapping out individual elements or shifting them around between regions of a page. And libraries of reusable content elements become the norm — so that you’re now picking your images, videos, donate banners and more from lists of existing content or existing content styles, instead of trying to hunt down how you did something similar the last time.

Structuring your content into individual, reusable elements drives a lot of collateral benefits:

• Ease of content creation. Authoring with a consistent set of components lets you focus more on content, hierarchy, and intent — instead of trying to remember how you got a video in there at all the last time.
• Styling and restyling. Using discrete elements of content means that your HTML markup and CSS remains consistent across all of your various pages. When the time comes to update the look & feel of your website, changes made on one article page will “just work” on the rest of your articles and all of their elements.
• Content migration. The day might arrive when you consider migrating your content to a newer platform or different software. Structured content makes that a snap: Each image, video, slideshow, or paragraph of text gets individually ported to its new home on the new platform. On the contrary, migrating a mess of markup, tokens, and style codes stored in a single body field means writing complex, custom code to recognize those items and deal with them appropriately — not a simple task.
• Syndication. Want to feature your content in a collection via RSS or deploy it to an app with XML? With structured content apps and other websites can consume just the elements they want of your content (maybe the first paragraph, a title, an image, or a link) then display those elements appropriately according to their own styles — rather than just grabbing a few hundred lines of a single, messy body field and making do.
• Libraries, lists, and cross-promotion. Want to see all the slideshows that appear in articles with a specific tag? Maybe a list of pull-quotes from your most recent blog posts? Structured content creates a world of opportunity around libraries, lists and cross-promotions.

Structuring content facilitates reuse, and it positions your content to go on living in a variety of platforms or a variety of presentations — long after you first click the Publish button.

The two faces of web content

Structuring content is an essential part of beautiful, intuitive digital authoring, but it’s only one side of the coin. The other side is putting an end to the two faces of your web content. Excepting some of the advances of modern WYSIWYG editors, web content has always had two very different faces: View and Edit.

The way content is edited and the way it’s presented are often strikingly different. Especially if you’re still holding on to “one giant body field” innovations like custom tokens, BBCode, Markdown, or short codes, it can take a handful of Preview to Edit to Preview roundtrips to get your content exactly how you want it. Even if you’ve made the move to structured content, a majority of web-based editors are marked by a dramatic difference between the “back end” and the “front end” — just one more digital convention getting between inspired authors and publishing beautiful content easily.

The king is dead, long live the king

The body field is dead. It’s taken center stage in publishing digital content for long enough. Plenty of web products (Medium and Notion, to name a couple) have driven nails in that coffin for publishers creating content on those specific platforms. But what about content creators and web editors working within their organizations’ websites, in custom web applications where content authoring is just one of several important features? What about you, and your organization’s website? What’s next for your web application?

For Drupal websites, we think Layout Paragraphs is what’s next. It was first released to the Drupal community nearly a year ago, and has been the beneficiary of ambitious development ever since.

The Layout Paragraphs module makes it dead simple for authors to create robust, multimedia content from a library of available components. It offers intuitive drag-and-drop controls for managing content flow and layout, and with appropriate styling can bring the two faces of web content — View and Edit — closer than ever before. We built Layout Paragraphs to embrace the future of multimedia content authoring and to solve the problems we watch clients work through every day.

You can watch a short, two-minute demo of Layout Paragraphs here, or follow the instructions in that post to see it in action for yourself.

James Nettik

Dictionaries are one of the most frequently used data structures in python. It contains data in the form of key value pairs. While processing the data with dictionaries, we may need to iterate over the items in the dictionary to change the values or read the values present in the dictionary. In this article, we will see various ways for iterating over dictionary in python.

Iterating over a dictionary using for loop

As we iterate through lists or tuples using for loops in python, we can also iterate  through a python dictionary using a for loop.

When we try to iterate over a dictionary using for loop,it implicitly calls __iter__() method. The __iter__() method returns an iterator with the help of which we can iterate over the entire dictionary. As we know that dictionaries in python are indexed using keys, the iterator returned by __iter__() method iterates over the keys in the python dictionary.

So, with a for loop, we can iterate over and access all the keys of a dictionary as follows.

myDict={"name":"PythonForBeginners","acronym":"PFB","about":"Python Tutorials Website"} print("The dictionary is:") print(myDict) print("The keys in the dictionary are:") for x in myDict: print(x)

Output:

The dictionary is: {'name': 'PythonForBeginners', 'acronym': 'PFB', 'about': 'Python Tutorials Website'} The keys in the dictionary are: name acronym about

In the output we can see that all the the all the keys have been printed. In the for loop the iterator x iterates over all the keys in the dictionary which are then printed.

Having obtained the keys of the dictionary using for loop, we can also iterate over the values in the dictionary as follows.

myDict={"name":"PythonForBeginners","acronym":"PFB","about":"Python Tutorials Website"} print("The dictionary is:") print(myDict) print("The values in the dictionary are:") for x in myDict: print(myDict[x])

Output:

The dictionary is: {'name': 'PythonForBeginners', 'acronym': 'PFB', 'about': 'Python Tutorials Website'} The values in the dictionary are: PythonForBeginners PFB Python Tutorials Website

In the code above, we have simply obtained an iterator which iterates over the keys in the and then we have accessed the values associated to the keys using the syntax dict_name[key_name] and then the values are printed.

Iterate over keys of a  dictionary

We can use the keys() method to iterate over keys of a dictionary. The keys() method returns a list of keys in the dictionary when invoked on a dictionary and then we can iterate over the list to access the keys in the dictionary as follows.

myDict={"name":"PythonForBeginners","acronym":"PFB","about":"Python Tutorials Website"} print("The dictionary is:") print(myDict) print("The keys in the dictionary are:") keyList=myDict.keys() for x in keyList: print(x)

Output:

The dictionary is: {'name': 'PythonForBeginners', 'acronym': 'PFB', 'about': 'Python Tutorials Website'} The keys in the dictionary are: name acronym about

We can also access values associated with the keys of the dictionary once we have the keys in the list as follows.

myDict={"name":"PythonForBeginners","acronym":"PFB","about":"Python Tutorials Website"} print("The dictionary is:") print(myDict) print("The values in the dictionary are:") keyList=myDict.keys() for x in keyList: print(myDict[x])

Output:

The dictionary is: {'name': 'PythonForBeginners', 'acronym': 'PFB', 'about': 'Python Tutorials Website'} The values in the dictionary are: PythonForBeginners PFB Python Tutorials Website Iterate over values of a dictionary in python

If we only want to access the values in the dictionary, we can do so with the help of the values() method.The values() method when invoked on a dictionary returns a list of all the values present in the dictionary. We can access the values in the dictionary using values() method and for loop as follows.

myDict={"name":"PythonForBeginners","acronym":"PFB","about":"Python Tutorials Website"} print("The dictionary is:") print(myDict) print("The values in the dictionary are:") valueList=myDict.values() for x in valueList: print(x)

Output:

The dictionary is: {'name': 'PythonForBeginners', 'acronym': 'PFB', 'about': 'Python Tutorials Website'} The values in the dictionary are: PythonForBeginners PFB Python Tutorials Website

In the output, we can see that all the values present in the dictionary are printed one by one.

Iterating over items in a dictionary in python

We can iterate over and access the key value pairs using the items() method. The items() method when invoked on a dictionary, returns a list of tuples which have keys and values as pairs. Each tuple has a key on its 0th index and the value associated with the key is present on the 1st index of the tuple. We can access the key value pairs using items() method as shown below.

myDict={"name":"PythonForBeginners","acronym":"PFB","about":"Python Tutorials Website"} print("The dictionary is:") print(myDict) print("The items in the dictionary are:") itemsList=myDict.items() for x in itemsList: print(x)

Output:

The dictionary is: {'name': 'PythonForBeginners', 'acronym': 'PFB', 'about': 'Python Tutorials Website'} The items in the dictionary are: ('name', 'PythonForBeginners') ('acronym', 'PFB') ('about', 'Python Tutorials Website')

Along with iterating the items in the dictionary using the items() method,we can also iterate over keys and values of  a dictionary using two iterators in the for loop as follows.

myDict={"name":"PythonForBeginners","acronym":"PFB","about":"Python Tutorials Website"} print("The dictionary is:") print(myDict) itemList=myDict.items() print("The key value pairs in the dictionary are:") for x,y in itemList: print(x,end=":") print(y)

Output:

The dictionary is: {'name': 'PythonForBeginners', 'acronym': 'PFB', 'about': 'Python Tutorials Website'} The key value pairs in the dictionary are: name:PythonForBeginners acronym:PFB about:Python Tutorials Website

In the above program, the first iterator iterates over the keys in the dictionary and the second iterator iterates over the respective values associated with those keys which are present in the form of tuple containing key value pairs in the list returned by items() method. 

Conclusion

In this article, we have seen various ways to iterate over the data in dictionaries. We have seen how to access the keys and values of the dictionary using for loop and inbuilt methods like keys(), values() and items(). We can also write the programs used in this article with exception handling using python try except to make the programs more robust and handle errors in a systematic way. Stay tuned for more informative articles.

The post Iterating over dictionary in Python appeared first on PythonForBeginners.com.

Spam.

This word contains so much pain and headache for any site owner. It is not possible to 100% eradicate spam from your website. However, it is possible to reduce these rates to a minimum, provided that you know how to prevent website spam.

Today we are going to tell you about the best ways to stop website spam. Our web maintenance company has chosen 6 handy anti-spam Drupal modules. You have to choose the one that suits you best and add it to the site.

During DrupalCon North America 2021, the members of the Drupal Community Working Group announced the winner of the 2021 Aaron Winborn Award, AmyJune Hineline (volkswagenchick).  

Since joining the Drupal community in 2015 after a career change, AmyJune has made an impact in multiple aspects of the community. She is a core mentor, helps organize multiple Drupal events around North America, and is a tireless advocate for accessibility and inclusivity in the community. In addition, she is one of the organizers of A11yTalks, a member of the Community Health Team, a documentation contributor, and has been credited in over 700 issues in the past year. Much of her work is sponsored by her employer, Kanopi Studios. 

Multiple people nominated AmyJune for this award. Here are some of the things they said:

"AmyJune has worked tirelessly for years to encourage contribution to the Drupal project from people with ALL levels of experience and skill sets. She creates a welcoming environment for anyone interested in contributing and makes the process less intimidating."

"She shares her knowledge freely and frequently, and also shares prolifically on social media to help others spread the word about their community work and celebrate successes."

"AmyJune is dedicated to the Drupal project on a level I have personally never seen before."

"AmyJune encourages accessibility and inclusiveness in all of her interactions within the community."

"I've never seen someone more dedicated to community and especially the Drupal community."

"I'm new to the Drupal Community (I joined 5 months ago), and I'm also new to the tech industry, and of course, the onboarding with no coding skills whatsoever has been quite a challenge. In this brief period, AmyJune has been an inspiring role model to look up to."

This year, there were 29 individuals nominated for the award. In the coming weeks, the CWG will be contacting all nominees to let them know of their nomination and thank them for their continued work in the community.

In addition to the physical award shipped to AmyJune, she was also provided with a free ticket to DrupalCon Global which she graciously donated to a deserving community member. The physical award that was hand-crafted by Drupal community member Bo Shipley (simplyshipley).  

The award is named after a long-time Drupal contributor who lost his battle with ALS in 2015. This award recognizes an individual who, like Aaron, demonstrates personal integrity, kindness, and an above-and-beyond commitment to the Drupal project and community. Previous winners of the award are Cathy Theys, Gabór Hojtsy, Nikki Stevens, Kevin Thull, Leslie Glynn, and Baddý Breidert. Current CWG members, along with previous winners, selected the winner based on nominations submitted by Drupal community members.

Nominations for the 2022 award will open in early 2022.
 

Introduction to Radix Sort

The radix (or base) is the number of digits used to represent numbers in a positional numeral system. For the binary system, the radix is 2 (it uses only two digits - 0 and 1). For the decimal system, the radix is 10 (it uses ten digits to represent all numbers - from 0 to 9).

A positional numeral system is, in simple terms, a number writing system, where the weight (or the value) of a digit is determined by its position. For example, in the number 123, 1 has more value than 3 because it's in position that denotes hundreds, and the 2 is in the tens.

Radix Sort can be used to lexicographically sort many types of data - integers, words, emails, but is mainly used to sort collections of integers and strings (that are mapped to appropriate integer keys).

It's a non-comparative sorting algorithm, meaning that it doesn't sort a collection by comparing its individual elements, but rather uses the inherent nature of the data its sorting to sort faster - it sorts data based on their radix.

Comparative sorting algorithms have the best case time complexity of O(nlogn), which is comparatively worse to linear execution time (O(n+k)) of non-comparative algorithms.

For example, let n be the number of elements to be sorted, and k is the range of allowed element values.

Counting Sort (a popular non-comparative algorithm) has the complexity of O(n+k) when the k is in the range from 1..n. But, if elements range from 1..n², then the complexity rises to O(n²), which is worse than any comparative sorting algorithm.

Counting Sort has the potential to be significantly faster than other popular comparative algorithms, though, only if a certain condition was fulfilled.

The idea of the Radix Sort is to upgrade Counting Sort so that it maintains the linear time complexity even if the range of elements' values drastically exceeds the number of elements.

In fact, Radix Sort inherently uses Counting Sort as the main subroutine, with a few tweaks to overcome the issues that arise with an incrased range of elements' values.

• Counting Sort Algorithm
• Why Use Counting Sort?
• How does Counting Sort work?
• Implementing Counting Sort
• Counting Sort Complexity
• Radix Sort Algorithm
• Implementing Radix Sort
• Radix Sort Complexity

Counting Sort Algorithm

In order to get a grasp of Radix Sort, we'll have to delve into Counting Sort first, implement it and observe the downfall with an increased number of element values.

Why Use Counting Sort in the Radix Sort?

Counting sort is a stable, non-comparative sorting algorithm, and it is mainly used to sort integer arrays. All of these characteristics are important for its use in Radix Sort. You can use other algorithms as the subroutine, as long as they have these characteristics, though, Counting Sort is the most natural matchup.

Radix Sort needs to maintain a relative order of elements with the same key values in the input array while sorting the same place value digits, therefore, our main subroutine by definition needs to be some sort of stable sorting algorithm:

Non-comparative sorting algorithms generally have linear complexity, so they will have less impact on the complexity of the Radix Sort.

How Does the Counting Sort Work?

Let's take a look at an unsorted integer array, which we'll sort using Counting Sort:

I = [2, 2, 0, 6, 1, 9, 9, 7]

Counting Sort works by counting the number of elements, that fit a distinct key value, and then calculates the positions of each key.

First of all, we'll find the maximum element in the input array - max = 9.

Then, we'll create an auxiliary array with max+1 elements. This is the count array (C), which will be used to store the number of occurrences of each element in the input array.

Initially, all counts are initialized to 0:

C = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] # Count array #indices: 0 1 2 3 4 5 6 7 8 9

Now, we need to go through the following steps:

1. Traverse the input array and increase the corresponding count for every element by 1

For example, if we come across an element with the value of 2 in the input array (I), we add 1 to the element with the index 2 in the count array:

I = [2, 2, 0, 6, 1, 9, 9, 7] # The first element is 2 ^ C = [0, 0, 1, 0, 0, 0, 0, 0, 0, 0] # We increase count of 2nd element by 1 #indices: 0 1 2 3 4 5 6 7 8 9

After this step, the count array will store the number of occurrences of each element in the input array:

C = [1, 1, 2, 0, 0, 0, 1, 1, 0, 2] #indices: 0 1 2 3 4 5 6 7 8 9 # Element 0 has 1 occurrence # Element 1 has 1 occurrence # Element 2 has 2 occurrences # Element 3 has no occurrences...

2. For each element in the count array, sum up its value with the value of all its previous elements, and then store that value as the value of the current element:

C = [1, 2, 4, 4, 4, 4, 5, 6, 6, 8] #indices: 0 1 2 3 4 5 6 7 8 9 # Element 0 = 1 # Element 1 = 1 + 1 # Element 2 = 1 + 1 + 2 # Element 3 = 1 + 1 + 2 + 0 #...

This way, we are storing the cumulative sum of the elements of the count array, on each step.

3. Calculate element position based on the count array values

To store this sorted sequence, we'll need to create a new array. Let's call it the output array (O), and initialize it with k zeros, where k is the number of elements in the input array:

O = [0, 0, 0, 0, 0, 0, 0, 0] // Initialized output array #indices: 0 1 2 3 4 5 6 7

For each element I[i] (starting from the end) in the input array:

1. Find the index in the count array that is equal to the value of the current element I[i]
   • That's the element C[j] where j=I[i]
2. Subtract 1 from the value of the C[i]
   • Now we have newValue = C[i]-1
3. Store the I[i] to the O[newValue]
4. Update the C[i] with the newValue

In the end, the output array contains the sorted elements of the input array!

Implementing Counting Sort in Python

Now, with all that out of the way - let's go ahead an implement Counting Sort in Python:

def countingSort(inputArray): # Find the maximum element in the inputArray maxEl = max(inputArray) countArrayLength = maxEl+1 # Initialize the countArray with (max+1) zeros countArray = [0] * countArrayLength # Step 1 -> Traverse the inputArray and increase # the corresponding count for every element by 1 for el in inputArray: countArray[el] += 1 # Step 2 -> For each element in the countArray, # sum up its value with the value of the previous # element, and then store that value # as the value of the current element for i in range(1, countArrayLength): countArray[i] += countArray[i-1] # Step 3 -> Calculate element position # based on the countArray values outputArray = [0] * len(inputArray) i = len(inputArray) - 1 while i >= 0: currentEl = inputArray[i] countArray[currentEl] -= 1 newPosition = countArray[currentEl] outputArray[newPosition] = currentEl i -= 1 return outputArray inputArray = [2,2,0,6,1,9,9,7] print("Input array = ", inputArray) sortedArray = countingSort(inputArray) print("Counting sort result = ", sortedArray)

Running the code above will yield us the following output:

Input array = [2, 2, 0, 6, 1, 9, 9, 7] Counting sort result = [0, 1, 2, 2, 6, 7, 9, 9] Counting Sort Complexity

The time complexity of the counting sort is O(n+k), where n is the number of elements in the input array, and k is the value of the max element in the array.

The problem occurs when the value of the largest element drastically exceeds the number of elements in the array. As the k approaches n², the time complexity gets closer to O(n²), which is a horrible time complexity for a sorting algorithm.

This is where Radix Sort kicks in.

Radix Sort Algorithm

Instead of counting the elements by their distinct key value - Radix Sort groups digits by their positional value and performing Counting Sort in each group. The starting position can vary - LSD (Least Significant Digits) or MSD (Most Significant Digits) are two common ones, and accordingly, these variations of Radix Sort are called LSD Radix Sort and MSD Radix Sort.

Let I = [2, 20, 61, 997, 1, 619] be the input array that we want to sort:

We'll focus on LSD Radix Sort.

Radix Sort Algorithm

The steps taken by Radix Sort are fairly straighforward:

1. Find the maximum element in the input array - max = 997
2. Find the number of digits in the max element - D = 3
3. Initialize the place value to the least significant place - placeVal = 1
4. For D times do:
   1. Perform the counting sort by the current place value
   2. Move to the next place value by multiplying placeVal by 10

Implementing Radix Sort in Python

And finally, with that out of the way, let's implement Radix Sort in Python:

def countingSortForRadix(inputArray, placeValue): # We can assume that the number of digits used to represent # all numbers on the placeValue position is not grater than 10 countArray = [0] * 10 inputSize = len(inputArray) # placeElement is the value of the current place value # of the current element, e.g. if the current element is # 123, and the place value is 10, the placeElement is # equal to 2 for i in range(inputSize): placeElement = (inputArray[i] // placeValue) % 10 countArray[placeElement] += 1 for i in range(1, 10): countArray[i] += countArray[i-1] # Reconstructing the output array outputArray = [0] * inputSize i = inputSize - 1 while i >= 0: currentEl = inputArray[i] placeElement = (inputArray[i] // placeValue) % 10 countArray[placeElement] -= 1 newPosition = countArray[placeElement] outputArray[newPosition] = currentEl i -= 1 return outputArray def radixSort(inputArray): # Step 1 -> Find the maximum element in the input array maxEl = max(inputArray) # Step 2 -> Find the number of digits in the `max` element D = 1 while maxEl > 0: maxEl /= 10 D += 1 # Step 3 -> Initialize the place value to the least significant place placeVal = 1 # Step 4 outputArray = inputArray while D > 0: outputArray = countingSortForRadix(outputArray, placeVal) placeVal *= 10 D -= 1 return outputArray input = [2,20,61,997,1,619] print(input) sorted = radixSort(input) print(sorted)

Running the code above will yield us the following output:

[2, 20, 61, 997, 1, 619] [1, 2, 20, 61, 619, 997] Radix Sort Complexity

As we stated before, Radix Sort has linear time complexity. If we use Counting Sort as the main subroutine, the complexity of radix sort is O(d(n+k)). That is because we are executing the counting sort d times, and the complexity of the Counting Sort itself is O(n+k).

Conclusion

Radix sort is a great sorting algorithm to use in some specific cases. Some benchmarks have even shown that radix sort can execute up to 3 times faster than other, more general-purpose sorting algorithms.

It shines when the input array has shorter keys, or the range of the element values is smaller. But has poor space complexity in other cases, when the range of element values is quite large and elements have too many digits in their representation.

That is the main reason why the radix sort is not as widely used as some other types of sorting algorithms, even if it has linear time complexity.

https://www.wsj.com/articles/amazon-strong-arms-partners-across-multiple-businesses-11618410439

"Amazon.com Inc. last year told smart-thermostat maker Ecobee it had to give the tech giant data from its voice-enabled devices even when customers weren’t using them."

"Amazon responded that if Ecobee didn’t serve up its data, the refusal could affect Ecobee’s ability to sell on Amazon’s retail platform..."

In this example, you will learn to select a random element from the list.

In Drupal, files can be uploaded to the site for users to view or download. This can be easily achieved by creating a file or image field on content types.

In the back end, a list of all the files uploaded can be viewed by the administrator, by going to Administration > Content > Files (admin/content/files).

Files uploaded can be easily removed from the individual content pages (see the image below), but removing them entirely from the system is another story. You might be surprised that you cannot find a button, a link or an option to remove these deleted files entirely from the system.

After deleting files on content, if you go to the Files (admin/content/files) page, you will find the deleted files are still there, and the status still shows ‘Permanent’, even though they are already removed from the nodes. It seems very confusing. Removing files from content and removing files from the system are two different things in Drupal.

To remove files from the system, we need to add the file delete function. This can be achieved by installing the File Delete module.

In this example, you will learn to read a file line by line into a list.

Season of KDE 2021 Status Report

This year I participated SoK as mentor for KWeather and Kalk. Now the work has been done, let me sum up the new features we bring to the projects.

KWeather

When the project name is KWeather, the actual code is committed to KWeatherCore, a Weather library. We decided to add weather alerts functionality to KWeatherCore, possibly easing the difficulty later if we want to develop a weather alerts daemon.

The data source is from metroelogy institutes around the world. Most of them provide CAP protocol for alerts. So the main goal is to write two parsers, one for CAP feed parsing and other for the CAP parsing. Anjani Kumar finished classes representing CAP feed entry and CAP message, as well as the CAP parser. Nikunj Goyal is supposed to work on feed parser and some of the daemon code. But due to various reasons, he wrote the AlertManager class which is an utility class. I wrote the feed parser and finished most of the documentation.

Since the lack of man hours, we weren’t able to finish the promised daemon. However the library is working as expected. We use config files (in json format) to avoid hardcoding different formats of CAP feed. So in the future we can support more countries without edit our code (and releasing new library version).

Kalk

Kalk is calculator for Plasma Mobile, this SoK we’re bringing binary mode to it.

Though as simple as the project name suggests, in the middle of the SoK we decided to switch from primitive double type to GMP and MPFR for our base type. Thankfully we’re using Bison/Flex for our math engine so swithing to multi-precision big number type doesn’t require use to rewrite Kalk again, wrapper classes still needs to be created. We re-used Knumber from KCalc. Since it’s a in-source library for KCalC, and lacking some helper functions Kalk needs, it has been copied to Kalk and modified since. Rohan Asokan wrote binary number type wrapper for Knumber (though later reverted because of preference for native GMP/MPFR binary <-> string converting functions), and math engine supports simple binary calculations. He also managed to add UI for binary calculator using Kirigami.

Date calculator was originally planned, but I decided to drop it eventually. Kalk is a simple calculator, I don’t want it to be bloat. Rohan Asokan did have a prototype for it though.

In this blog post we provide instructions on how to enable, configure and use the DWP consent pattern in a Drupal 8/9 based website.

In this example, you will learn to print output without a newline.

In this example, you will learn to get a substring of a string.

In this example, you will learn to get the last element of this list.