Planet Python

(Best viewed on a larger than phone screen).

 

Last year I blogged on a taking a problem and solving it in multiple ways, just for the fun of it.

Triplet sum  is a problem solved by many, and I had my own description of the problem so I decided to investigate it using ChatGPT and see what it said.

In the following I chose not to use screen grabs of the ChatGPT session, but instead copy text input as me 1 through 5, and responses as c 1 through 6. I use Python comments for my views on its responses.

Familiarise yourself with my earlier blog post to better understand the following interaction:

# -*- coding: utf-8 -*- """triplet_sum_chatgpt.py
Investigation of Chatgpt and its solution to:    http://paddy3118.blogspot.com/2022/10/riffing-on-triplet-sum-to-target.html
Created on Sun Feb 26 10:26:41 2023
@author: paddy"""
#%% BEGIN Chatgpt interaction
me1 = """Given a list of integers and a target integer, return all "normalised" tripletsof numbers from that list that sum to the target value.Normalised triplets are a tuple of three values in sorted order,i.e. (t0, t1, t2) where t0 <= t1 <= t2."""
c1 = """To solve this problem, we can start by sorting the given list of integers. Then,we can use a nested loop to iterate over all possible pairs of numbers from thelist, and for each pair, we can use binary search to find the third number thatwould make the sum equal to the target integer. We can then check if thetriplet is in sorted order and add it to the result if it is.
Here's the Python code that implements this approach:"""
def find_triplets(arr, target):    arr.sort()    n = len(arr)    res = []    for i in range(n - 2):        # skip duplicates        if i > 0 and arr[i] == arr[i-1]:            continue        j = i + 1        k = n - 1        while j < k:            s = arr[i] + arr[j] + arr[k]            if s == target:                # check if triplet is in sorted order                if j == i + 1 or arr[j] != arr[j-1]:                    res.append((arr[i], arr[j], arr[k]))                j += 1                k -= 1            elif s < target:                j += 1            else:                k -= 1    return res
c2 = """This code has a time complexity of O(n^2), where n is the length of the inputlist. The sorting step takes O(n log n) time, and the nested loop takes O(n^2)time. The binary search inside the loop takes O(log n) time, so the overalltime complexity is O(n^2 log n).
Note that this code assumes that there are no duplicates in the input list.If there are duplicates, we need to skip them in the loop to avoid generatingduplicate triplets. The code handles this by checking if the current number isequal to the previous number, and skipping it if it is."""
#%% My follow-up
# As in original blog post:def tests1(funcs, args=None):    if args is None:        args = [([2, 0, 1, 3], 4),                ([0,1,3,2,4,5,6], 11),                ([0,1,2], 6),                ([0,1,2], 3),                ([0,1,2], 1),                ([0,1], 2),                ([0,1], 1),                ([0], 0),                ([], 99),                ]    seperator, indent = ('---\n', '  ') if len(funcs) > 1 else ('', '')    lines = []    for numbers, target in args:        line = ''        for func in funcs:            line += f"{indent}{func.__name__}({numbers}, {target}) = "            try:                answer = func(numbers, target)                line += str(answer)            except Exception as ex:                line += str(ex)            line += '\n'        lines.append(line)    print(seperator.join(lines))
tests1([find_triplets])
# # The above prints the correct results of:# find_triplets([2, 0, 1, 3], 4) = [(0, 1, 3)]# find_triplets([0, 1, 3, 2, 4, 5, 6], 11) = [(0, 5, 6), (1, 4, 6), (2, 3, 6), (2, 4, 5)]# find_triplets([0, 1, 2], 6) = []# find_triplets([0, 1, 2], 3) = [(0, 1, 2)]# find_triplets([0, 1, 2], 1) = []# find_triplets([0, 1], 2) = []# find_triplets([0, 1], 1) = []# find_triplets([0], 0) = []# find_triplets([], 99) = []
#%% Time complexities of blog answers
me2 = """What is the time complexity of: def triplet_sum1(numbers: list[int], target: int) -> list[tuple[int]]:    "triplet-sum by filtering all triplet combinations"    return [triplet for triplet in combinations(sorted(numbers), 3)            if sum(triplet) == target]"""
c3 = """The time complexity of the triplet_sum1 function is O(n^3), where n is thelength of the input list.
The function uses the combinations function from the itertools module togenerate all possible triplets of numbers from the input list. The combinationsfunction generates all possible combinations of k elements from the input list,where k is the second argument to the function. In this case, k is 3, socombinations(sorted(numbers), 3) generates all possible triplets of numbersfrom the sorted input list. The number of possible triplets of n elements is nchoose 3, which is equal to (n * (n-1) * (n-2)) / 6. Therefore, thecombinations function generates O(n^3) triplets.
The generated triplets are then filtered using a list comprehension, whichchecks if the sum of each triplet is equal to the target integer. Checking thesum of a triplet takes constant time, so the filtering step takes O(n^3) time.
Therefore, the overall time complexity of the triplet_sum1 function is O(n^3).This is slower than the approach used in the previous answer, which has a timecomplexity of O(n^2 log n)."""
# The answer looks good. triplet_sum1 is written to obviousely generate the# correct output.

me3 = """what is the time complexity of: from itertools import islice
def triplet_sum3(numbers: list[int], target: int) -> list[tuple[int]]:    "Triplet-sum with short-circuited combinations and less copying of numbers."    numbers.sort()    answers = []
    for i, item in enumerate(numbers):        isum = item        if isum <= target:            for j, jtem in enumerate(islice(numbers, i+1, None), i+1):                jsum = isum + jtem                if jsum <= target:                    for k, ktem in enumerate(islice(numbers, j+1, None), j+1):                        ksum = jsum + ktem                        if ksum == target:                            answers.append((item, jtem, ktem))                        elif ksum > target:                            break                else:                    break # Skip as jsum > target        else:            break # Skip as item > target
    return answers"""
c4 = """The time complexity of the triplet_sum3 function is O(n^2), where n is thelength of the input list.
The function uses a nested loop to generate all possible triplets of numbersfrom the sorted input list. The outer loop iterates over each item in the inputlist, which takes O(n) time. The inner loop uses the islice function from theitertools module to iterate over the remaining items in the input list afterthe current item, which takes O(n) time. The innermost loop does the same forthe remaining items after the second item in the triplet, which again takesO(n) time.
Therefore, the overall time complexity of the function is O(n^3). However, thefunction uses short-circuiting to avoid iterating over unnecessarycombinations. If the sum of the first item and the second item in a triplet isgreater than the target, the function skips the remaining items in the innerloop and moves on to the next item in the outer loop. Similarly, if the sum ofthe first item is greater than the target, the function skips the remainingitems in the outer loop. This reduces the number of iterations of the innerloops and makes the overall time complexity O(n^2).
Note that the function sorts the input list before generating the triplets.Sorting takes O(n log n) time, which is dominated by the O(n^2) time complexityof the nested loops. Therefore, the overall time complexity of the function isO(n^2)."""
# The above is wrong, in that all triplets are not generated then tested as# it states in the second paragraph.# The calculated time complexity looks plausible.
# Let's try it with my final function that can do more than triplets
me4 = '''what is the time complexity of: def triplet_sum4(numbers: list[int], target: int, n_tuple: int=3) \        -> list[tuple[int]]:    """    Generate triplets from numbers that sum to target in sorted order.    Indexes the sorted numbers without otherwise copying them.
    n_tuple=3 for triplets, 2 for pairs, etc, for any +ve int    """    numbers.sort()    n_len = len(numbers)    answers = []    indices = [-1] * n_tuple  # indices for triples; indices[0] must be -1    incrementing = 0          # index to increment at any time    accum = [None] * n_tuple  # (partial) sums in accumulator
    while True:        indices[incrementing] += 1        if indices[incrementing] >= n_len:            # maxed out this index so go back one            incrementing -= 1            if incrementing < 0:                break            continue  # continue incrementing 'outer' index        if incrementing < n_tuple - 1:            indices[incrementing+1] = indices[incrementing]            acc = accum[incrementing] = numbers[indices[incrementing]] + (                                    0 if incrementing == 0                                    else accum[incrementing - 1])            if acc > target:                # maxed out this index so go back one                incrementing -= 1                if incrementing < 0:                    break                continue
            incrementing += 1  # next index        elif incrementing == n_tuple - 1:            # updated last index            acc = numbers[indices[incrementing]] + (                    0 if incrementing == 0                    else accum[incrementing - 1])            if acc == target:                answers.append(tuple(numbers[i] for i in indices))    return answers'''
c5 = """The time complexity of the triplet_sum4 function is O(n^(k-1)), where n is thelength of the input list and k is the size of the tuple being generated.
The function generates tuples by incrementing indices into the sorted inputlist. It uses a single loop that iterates until all tuples of the desired sizehave been generated. Each iteration increments one of the indices and updatesan accumulator array that keeps track of the sum of the numbers indexed so far.If the sum of the numbers indexed so far exceeds the target, the functionbacktracks to the previous index and continues incrementing from there. If thelast index is incremented and the sum of the numbers indexed equals the target,the function adds the tuple to the list of answers.
The time complexity of each iteration of the loop is O(n), since each index canbe incremented at most n times before the loop terminates. Therefore, the timecomplexity of generating all tuples of size k is O(n^k).
In the case of generating triplets (k=3), the time complexity is O(n^2), whichis lower than the time complexity of the triplet_sum3 function (also O(n^2))because the triplet_sum4 function generates fewer tuples. For larger values ofk, the time complexity of triplet_sum4 can become very large, and it may bemore efficient to use a different algorithm that avoids generating all possibletuples."""
# Its description of the workings of triplet_sum4 is wrong too!# All possible tuples are not generated then tested - it short circuits tuple# generation when it knows it will exceed the target.# I am unsure of its complexity calculation.
# I like that it compared, (one), previous answers.
# %% What's fastest?
me5 = """for finding triplets from the integers in range zero to 665 inclusive, that sumto six; which function is the fastest?"""
c6 = """To find the fastest function for finding triplets from integers in range zeroto 665 inclusive that sum to six, we need to test the performance of eachfunction and compare their execution times. Here's an example test code:

import timeit
def test_function(func):    numbers = list(range(666))    target = 6    result = func(numbers, target)    assert result == [(0, 2, 4), (0, 3, 3), (1, 1, 4), (1, 2, 3)]    print(f"{func.__name__} took {timeit.timeit(lambda: func(numbers, target), number=1000):.6f} seconds")
test_function(triplet_sum1)test_function(triplet_sum2)test_function(triplet_sum3)test_function(triplet_sum4)

This code creates a list of integers from 0 to 665, sets the target value to 6,calls each function with the same arguments, and checks if the output iscorrect. Then, it measures the execution time of each function with 1000iterations and prints the result.
Note that the execution times may vary depending on the machine and the versionof Python used.
Based on the results of the test, the fastest function is likely to betriplet_sum4, followed by triplet_sum3, triplet_sum2, and triplet_sum1,in that order. However, the performance difference between the functions isrelatively small for this input size, and other factors such as codereadability and maintainability may also be important considerations."""
# Whoops! The context established integers are used once from the input array.# In c6 result assumes input integers can be used twice, when they appear only# once in the input!
# Functions from prior context are first its generated `find_triplets`, then my# `triplet_sum` 1, 3, and 4. This last answer invents triplet_sum2 and forgets# its own find_triplets.

Conclusion

ChatGPT gives no feedback on its level of certainty. It will be wrong with the same authority as its truer statements, Some errors can be caught, but others will not be if you are relying on Chatgpt to be more knowlegeable than yourself.

 

Matt Harrison asked in a tweet what my programming history was, so here's a list of the programming languages I've learned.

AppleBasic (ancient history)

My first computer was an Apple ][+. I learned AppleBasic from a book that came with the computer. I wrote oodles of text-based games. The most popular was a Star Trek battle clone that was well-appreciated in 10th grade.

FoxPro (1997-2000)

I learned FoxPro at my first professional software development job. I liked FoxPro for DOS and still have a soft spot for CLI dbase-style languages. The same can't be said for Visual FoxPro for Windows, which never clicked for me.

JavaScript (1998+)

Originally I just copy/pasted scripts from sites and books. I struggled with it and the DOM until encountering JQuery in 2008 plus reading "JavaScript: The Good Parts" the same year. That helped and I learned to enjoy the language.

Perl (1999)

I spent about six months coding with Perl. I enjoyed it, especially the regular expressions that have served me so well over the years. The challenge was that in my self-teaching as a junior developer, I didn't think about maintainability. When I had to go back to maintain stuff I wrote I was lost, got frustrated, and left it behind. Probably not fair to Perl, but it is what it is.

ColdFusion (2000-2006)

This was popular in US Federal government circles. In a way, it was like a closed-source PHP, with lots of idiosyncrasies. The biggest problem the language had was people - for whatever reason it attracted those who coded via cargo cult copy/paste programming and generally bad software design who refused to change or adapt. Which sucked because coming in to maintain someone else's code was a nightmare. Still, I have a soft spot for ColdFusion because it helped me identify that I preferred more dynamic languages over static ones.

VBScript (2000-2001)

This was one of Microsoft's abortive attempts to hijack JavaScript. I liked it more than the JavaScript of the time, but as it only worked on Internet Explorer that limited its usefulness.

Java (2000-2006)

Java during the time I used it was designed to be for the "Enterprise". Which meant tons of boilerplate code, lots of XML, and a general lack of fun. The heavyweight of Enterprise Java was a huge turnoff. I struggled with it so much that I developed an intolerance for strongly typed languages (from which I've only recovered in the last few years). I admit my opinion of Java is biased, I'm sure it would be awesome if I dug into the modern version.

Python (2005+)

This is the programming language and community that changed my life. Python fits into my brain and it's my go-to language for all kinds of scripts.

Lua (2011-2012)

For a brief time, I did Lua and enjoyed it. If I were into game scripting I'd probably still be using it.

Modern NodeJS (2018+)

In 2018 I finally knuckled down and learned the ways of NodeJS. Much as I love Python, knowing NodeJS has been a game-changer for me. Arrow functions are fun and I wish Python had a decent analog.

C# (2020+)

In 2020 I started to play with the Unity game engine. C# clicked for me surprisingly fast and it was my favorite part of writing games. It's part of why I wonder if I should revisit Java since the languages are so similar.

Go (2020)

I did some Golang in late 2020. I liked it and saw the virtues, especially the concurrency model. I just don't have any use cases for it that my other tools don't solve.

JSX (2021+)

To build out a website for work I learned the fundamentals ReactJS and JSX over a weekend. This blog is my first project, which I continue to extend and modify. I like noodling with JSX, and I can see why it has become so popular. While arguably not a programming language, it's clear it is a huge productivity boost for me and others. It's fun to write and that's what counts.

TypeScript (2022+)

Kicking and screaming I've been dragged into the TypeScript world. I know professional FE devs love it, but I'm more partial to NodeJS because I feel like I'm coding, not playing with types. That said, it is kind of relaxing figuring out the types. Where I think TypeScript fails is I haven't seen any dominant libraries built around its typing features. For example, in Python we have pydantic and FastAPI, which are powered by type annotations. I haven't seen anything like that for TypeScript. If you know of anything like it, point me to it!

Rust (2022+)

In December of 2022, I started to learn Rust. Its use in WASM is fascinating, as is the performance boost and the sophistication of the compiler. If only I had a business reason to practice it, but for now it's just a hobby tool.

Summary

Many of the agile tools, principles and guidelines used by large IT enterprises can be used by small teams or individual engineers to improve efficiency and create more value.

In this article, I’d like to explore these concepts with you to help you level up your engineering practice.

The underlying agile principle I’ll explore is “Stop Starting. Start Finishing.” In other words, rather than constantly starting new work, concentrate on completing the work you are currently doing-transforming from a Starter to a Finisher.

When Starters think of a new idea, they drop what they are doing and start working on it while it’s fresh in their minds. Inevitably this leaves them with a trail of unfinished projects. Whereas Finishers complete the project they are working on before choosing their next project.

And this doesn’t just apply to building apps. If you learn to be a Finisher, you can apply this principle to all those To-Do tasks cluttering your mind.

Starter or Finisher?

Like most psychological traits, a Finisher mindset isn’t binary – it’s a continuum and it’s situational. But we all generally trend towards Starter or Finisher. Which are you?

To find out which way you trend, crack open your favorite editor and create two headings:

• Work I Started
• Work I Completed

Work I Started

Think back over the last 7 days. Jot down all the work you started. Take a few minutes to do this and be generous with your definition of work. It should include all the little chores you started.

Did you wash the car? Take the dog for a walk? File your taxes? Celebrate the sheer variety of all you do!

When you are finished with this list, congratulate yourself. You acted like a Finisher and completed a task.

Work I Completed

Now write down the work you fully completed. Be honest. Completed means fully completed to the standard you set when you set out to do it.

Results

Now for some simple data analysis. Count the number of tasks you started and the number you finished. Did you finish more than 80% of the tasks you started? If so – and this was a typical 7 day stretch for you – move right along! You are already a Finisher and you don’t need this article.

Still reading? Me too! Don’t beat yourself up. You are where almost all of us are. You tend towards being a Starter. But with a little change in attitude and by adopting a few tools you can become more of a Finisher.

By the way, if you looked at your lists and thought, “I could write simple Python app to analyze this data much more efficiently”, opened an IDE and started coding away, you have a brilliant engineering mindset, but you are not winning any contests for being a Finisher!

Are we wired like this?

The tendency to be distracted is rooted in the traits that kept us alive in ancient caves and jungles. A mind that focusses on a single thing and ignores new stimulus is not well suited to an environment where new opportunities like food sources or challenges like predators emerge constantly and unexpectedly.

When something new appears, we are hard-wired to drop what we are doing and concentrate on the new. But we can consciously overcome these instincts.

You too can be a finisher

Like most of you, I am a natural Starter, not a Finisher. As a scrum-master in engineering, I knew I had to change that. My journey involved adopting some brilliant ideas and awesome tools borrowed from agile ways of working that have proven themselves time and again.

Organize Your Work Backlog (A better To-Do list)

The first engineering principle we can draw on will help us organize our work. Replace that nebulous, ever shifting To-Do list with a backlog. This is the way most agile teams organize their work to get the most out of limited resources.

A backlog is simply a list of tasks you want to do some time in the future. It can be housed on a notepad, a whiteboard or as physical cards in a pile, or it can be kept in a digital tool. I installed Trello on my phone. I use it because it goes wherever I go.

It’s the guidelines you set for yourself in managing your backlog that make it work. Backlog guidelines evolve to suit you but here are my ground rules as a starting point-

• When you get a new task, add it to the backlog. The initial entry can be brief (you don’t want to interrupt what you are doing) and it should describe the task just enough so you will understand it later.
• Organize your backlog regularly-on a fixed schedule if possible. If you can set aside time daily to do this, it helps. In agile we call this refinement and it includes:
  • Fill in the tasks’ details so you always know what you wanted to do and why.
  • Prune your backlog. Get rid of stale tasks that you will never get round to.
  • Right size the tasks. Tasks that can be done in less than an hour are less daunting and more likely to get done.
  • Reorder the tasks by priority. Are any of them time sensitive? Some are almost certainly more important than others. While you do this, you may find some aren’t important at all and you can delete them.

This does take a little work (less than it sounds), but once you have a backlog that you frequently refine you will have solved those issues with your unorganized To-Do list, and you have the first step of a workflow – which we’ll discuss now.

Workflow

Now we have a backlog, how does all the work get done?

Tasks move out of the backlog through one or more states until they end up in a column called Done.

While a workflow can be as elaborate as you like, I suggest starting with a very simple workflow. I have 3 states work can be in.

• Backlog: (I hope you know what this is now. You can call it To Do if that helps)
• In Progress: No longer in your backlog. You are working on it.
• Done: The state all tasks aspire to. Your job is to help them get there!

The workflow works because of a few more guidelines:

• New tasks are only added to the backlog. Try not to start work that isn’t in the backlog because the backlog is your refined, prioritized list. If you do need to add an emergency task, you can put it into In Progress but try to limit that. If you keep adding work In Progress you are probably still being a Starter.
• Try to take work from the top of the backlog first since you agreed with yourself that this is your most important work.
• Limit how many tasks are In Progress to a definite number (3 is the number that works for me). This is covered in the Limit Work In Progress (WIP) section below.
• Only work on the task as it is described. Ideas tend to grow more elaborate and complex if you let them. Don’t let that happen. If you think of improvements as you work on a task, add these improvements as new backlog items-don’t do them now.
• Don’t obsess. If something will only take 5 minutes, you don’t need a card.
• If you need to break one of these guidelines every so often, don’t sweat it. Even if ‘run away from the escaped lion’ isn’t an In Progress card, you should probably do it anyway. If you are constantly breaking the guidelines, think about why that is and work out how to fix it.

Limiting Work In Progress (WIP)

If there is one takeaway from this post that is the most likely to help you become a ‘Finisher’ it’s this one. Limit Work In Progress (WIP).

Starters like us do too much at once. We just have so much going on that we feel we have to make progress on multiple fronts at once. But more-at-once kills productivity and Finishing.

If we tackle too many tasks we have to frequently context switch, incurring ‘startup costs’. This cost of switching tasks is real and proven. We pay for distractions, and we pay for trying to work on 5 things at once.

The engineering principle we can use to counteract the ‘too many things at once’ principle is limiting our Work In Progress (WIP) – a contract we make with ourselves. We choose a WIP limit and under normal circumstances we stick to it. This limit is the number of tasks we allow to be In Progress at once. This is so central to the way agile teams build software some of the work management tools like Jira allow you to set a WIP limit on each workflow column. If the team exceeds this number, the column goes scary red.

Just to be clear, a WIP of 3 does NOT mean I can work on an article, a Bite exercise and a project for my portfolio all at the same time, using 3 computers. I still only work on 1 task at once. The WIP is the number of tasks I have on my current list of tasks in progress. If I get stuck on one, I can switch to another.

NOTE: A WIP limit is a maximum but we don’t need to fill the maximum. Try to limit your actual number of tasks in progress to 1 even if you set a higher WIP limit. When you move a task to Done (look at you –being a Finisher) before you automatically move a task to In Progress from the backlog and start work, is there a task already In Progress you can finish first?

Other useful tools and techniques

Now, since the tasks we are working on are right sized and small (thank you, backlog refinement) we can use a couple of tools to help us focus and work on them to completion (thus becoming a Finisher).

My favorites are Pomodoro and micro-habits. Let’s see how they work together!

Pomodoro

Pomodoro focuses your attention on one In Progress task at a time. Coupled with backlog Management and your WIP limit, it is a powerful productivity tool.  It consists of these steps:

1. Decide on the task to be done (using the tools and techniques you now know about).
2. Set the pomodoro timer (typically for 25 minutes, related Pybites tool).
3. Work on the task. The discipline here is to focus on that single task. Avoid distractions. You may decide to turn off social media notifications and you could choose a work area without distractions.
4. End work when the timer rings and take a short break (typically 5–10 minutes). You may want to ‘just finish this task’ and postpone the break knowing you are only 5 minutes away from completion. But try to avoid that mindset. You are often not as near to done as you think, and there is solid science behind the cycle of work and breaks and the length of each.
5. If you have finished fewer than three pomodoros, go back to Step 2 and repeat until you go through all three pomodoros.
6. After three pomodoros are done, take the fourth pomodoro and then take a long break (typically 20 to 30 minutes). Once the long break is finished, return to step 2.

Since many software engineers have embraced Pomodoro and love to write fun and useful software, there are many online Pomodoro timers you can install / run online for free. You could even create your own.

What? Build it right now? NO! Break it down into tasks in your Backlog 

Micro Habits

Another concept to help you be a Finisher is that of micro habits. These train us to make our habits (which I relate to tasks) achievable in a short time. We will really accelerate our journey to Finishing by breaking tasks into small chunks – something we are likely to achieve in a single Pomodoro session. Micro habits become a self-fulfilling prophecy. We Finish frequently, so we expect to Finish, and this propels us to Finish more. Also, with smaller tasks we have less inertia to overcome when we think about starting a new one.

As a concrete example, let’s go back to those Pomodoro tracker programs we love to build. Could you write the entire tracker as a single Backlog item, move it to In Progress and finish it in a single Pomodoro session? I know I couldn’t. Using the micro habit principle, break this daunting task into smaller chunks, each of which you could likely achieve in a single session. Maybe the first task is to research and learn what a Pomodoro timer is? Or write code that just runs for 25 minutes and alerts you in some way? I think I could knock out a Python script that sleeps for 25 minutes and writes a line to output in a single Pomodoro session, so why don’t I define that as my first task and be a Finisher?

Minimum Viable Product (MVP)

The Minimum Viable Product (MVP) is another engineering principle that we can use to finish more work. The concept of an MVP is to build the absolute minimum product to satisfy your requirements. You can improve on it in later tasks. This mindset keeps you from adding scope to the work you are doing until it becomes a never-ending task, and each task you finish lets you get feedback on the next improvement.

As an example, if I want to create a means of getting from my home to a park, instead of working on a  helicopter rotor blade, could I start with the task of buying a bus pass? On the first rainy day where the bus is late, I will have feedback what to do next. Buy an umbrella? I may end up with that helicopter, but I will incrementally improve my experience in a way that 10 years of construction in my garage will not.

By the way, many of us in the software delivery world now refer to the Minimum Lovable Product (MLP) as an evolution from MVP. There are many MLP’s in production!

Inspect And Adapt

Another essential agile principle is Inspect and Adapt. Frequently look at WHAT you produced and HOW you produced it to see if you can make small improvements in the way you work that build over time.

Are you Finishing more? Learn from the work you didn’t finish. Be honest with what went wrong. Was the task small enough? Did you let it grow larger as you worked on? Were you distracted or did you start to do other work?

As you look into the reasons you were temporarily a Starter, not a Finisher, please be gentle on yourself. Yes, you want to learn what went awry so you can course correct, but this is not an opportunity to beat yourself up. You wouldn’t have stumbled if you hadn’t tried to walk.

Accountability partners / circles

Let’s face it. We stay more disciplined when we have to discuss our progress with a friend or group. Accountability works well if you regularly report on what you intend to do in the next 7 days and then discuss whether you accomplished that goal a week later. Please remember – you and your accountability partners should be cheerleaders for each other. Yay! I found an accountability partner on the Pybites slack channel. It’s a great place to start.

Related Pybites podcast episode.

Celebrate the wins

As you start to finish more, it’s important we celebrate those wins. Treat yourself for your successes. Try to make this a regular part of your relationship with your accountability partner or circle. If you celebrate winning and allow yourself to fail without consequence, you will create a virtuous circle where success leads to more success.

Related Pybites article.

Conclusion

As humans, we are alike in many respects. We are hard wired to concentrate on new shiny objects because they could be new food sources or dangers.

Translating this to the modern world, even when we fully committed to completing a task, we are likely to think of several other ideas that seem important enough to supersede what we are currently working on. But this context switching is stressful and inefficient. It’s better to Finish what we Started first, then decide what to work on next.

We explored this subtle attitude shift along with a few tried-and-tested agile tools and techniques to improve the amount of work we Finish.

On a personal note, I thought of the idea for this article a while back and created the outline. Before I finished writing it, I started a couple of fun new projects and the article lay here on my PC, unfinished. Undelivered. Wasted.

So I added this article to my backlog, concentrated on finishing my In Progress tasks and when the time was right (today) I committed to completing the article. I stayed focused and finished the initial draft.

Next, I will respond to feedback until it’s ready to publish. I just moved closer to being a Finisher, not a Starter. And you can too!

We use different file formats to store and transmit data generated from software systems. Sometimes, we need to convert an XML file to JSON to communicate between two software. This article discusses how to convert an XML file or string into JSON format in Python.

Table of Contents

1. What is XML?
2. What is JSON File Format?
3. XML String to JSON String in Python
4. Convert XML String to JSON File in Python
5. XML File to JSON String in Python
6. Convert XML File to JSON File in Python
7. Conclusion

What is XML?

XML stands for Extensible Markup Language. This is a standard markup language for describing and structuring data. XML was designed to store and transport data, and it is widely used in various applications and technologies, such as web services, database systems, document formats, and many others.

XML uses tags to define elements, attributes to define properties of those elements, and a hierarchical structure to organize the data. XML documents can be validated against a specific schema or Document Type Definition (DTD) to ensure their conformance to a specific standard.

For instance, the following is an XML string containing the details of an employee.

<?xml version="1.0"?> <employee> <name>John Doe</name> <age>35</age> <job> <title>Software Engineer</title> <department>IT</department> <years_of_experience>10</years_of_experience> </job> <address> <street>123 Main St.</street> <city>San Francisco</city> <state>CA</state> <zip>94102</zip> </address> </employee>

In the above XML string, <employee> is the root element. You can observe that each element in the string is represented using the following syntax.

<element_name> value </element_name>

XML is a platform-independent language. We can use it on any operating system and easily share it between different applications and systems. It is also human-readable and can be easily understood and edited by both machines and humans.

What is JSON File Format?

JSON stands for JavaScript Object Notation. It is a lightweight, text-based data interchange format that is easy for humans to read and write and easy for machines to parse and generate. Its structure resembles a python dictionary.

JSON is based on a subset of the JavaScript programming language, but it is language-independent and can be used with any programming language. It is often used as a data format for web APIs, as it is a common way to exchange data between the client and server.

JSON uses a simple syntax to represent data structures, such as objects and arrays. Data is represented using key-value pairs, where the keys are strings and the values can be of any JSON-supported data type, such as strings, numbers, objects, and arrays.

Following is the JSON representation of data shown in the previous XML string.

{ "employee": { "name": "John Doe", "age": 35, "job": { "title": "Software Engineer", "department": "IT", "years_of_experience": 10 }, "address": { "street": "123 Main St.", "city": "San Francisco", "state": "CA", "zip": 94102 } } }

JSON is more concise and easier to parse than XML, making it a popular choice for data transfer between applications and systems. It is also widely supported by modern programming languages, making it an ideal format for web-based applications.

To convert XML files and strings to JSON, we will use the xmltodict module and the json module.

XML String to JSON String in Python

To convert an XML string to a JSON string, we will first convert the XML string to a Python dictionary. For this, we will use the parse() method defined in the xmltodict module. The parse() method takes an XML string as its input argument and returns the corresponding dictionary.

Next, we will convert the python dictionary to a JSON string. For this, we will use the dumps() method defined in the json module. The dumps() method takes the python dictionary as its input argument and returns the corresponding JSON string after execution.

You can observe this in the following example.

import xmltodict import json xml_string="""<?xml version="1.0"?> <employee> <name>John Doe</name> <age>35</age> <job> <title>Software Engineer</title> <department>IT</department> <years_of_experience>10</years_of_experience> </job> <address> <street>123 Main St.</street> <city>San Francisco</city> <state>CA</state> <zip>94102</zip> </address> </employee>""" print("The XML string is:") print(xml_string) python_dict=xmltodict.parse(xml_string) json_string=json.dumps(python_dict) print("The JSON string is:") print(json_string)

Output:

The XML string is: <?xml version="1.0"?> <employee> <name>John Doe</name> <age>35</age> <job> <title>Software Engineer</title> <department>IT</department> <years_of_experience>10</years_of_experience> </job> <address> <street>123 Main St.</street> <city>San Francisco</city> <state>CA</state> <zip>94102</zip> </address> </employee> The JSON string is: {"employee": {"name": "John Doe", "age": "35", "job": {"title": "Software Engineer", "department": "IT", "years_of_experience": "10"}, "address": {"street": "123 Main St.", "city": "San Francisco", "state": "CA", "zip": "94102"}}} Convert XML String to JSON File in Python

Instead of converting it into a string, we can also convert an XML string to a JSON file. For this, we will use the following steps.

• First, we will convert the XML string to a dictionary using the parse() method defined in the xmltodict module.
• Then, we will open an empty JSON file in write mode using the open() function. The open() function takes the file name as its first input argument and the python literal “w” as its second argument. After execution, it returns a file pointer. 
• Once we get the file pointer, we will save the python dictionary as JSON to the file using the dump() method defined in the json module. The dump() method takes the dictionary as its first argument and the file pointer as the second input argument. After execution, it saves the dictionary into the json file as a json object.
• Finally, we will close the file using the close() method.

After executing the above steps, we can easily convert an XML string to a JSON file. You can observe this in the following example.

import xmltodict import json xml_string="""<?xml version="1.0"?> <employee> <name>John Doe</name> <age>35</age> <job> <title>Software Engineer</title> <department>IT</department> <years_of_experience>10</years_of_experience> </job> <address> <street>123 Main St.</street> <city>San Francisco</city> <state>CA</state> <zip>94102</zip> </address> </employee>""" python_dict=xmltodict.parse(xml_string) file=open("person.json","w") json.dump(python_dict,file) file.close()

The output file looks as follows.

JSON File XML File to JSON String in Python

To convert an XML file into a JSON string, we can use the dumps() method defined in the json module. For this, we will first open the XML file in read mode using the open() function. Then, we will read the contents of the XML file as a string using the read() method. 

We will use the following XML file to convert it into a JSON string.

XML File

Once we get the XML string, we will convert it to a python dictionary using the parse() method defined in the xmltodict module. Next, we will convert the dictionary into JSON string using the dumps() method defined in the json module. 

You can observe this in the following example.

import xmltodict import json xml_file=open("person.xml","r") xml_string=xml_file.read() python_dict=xmltodict.parse(xml_string) json_string=json.dumps(python_dict) print("The JSON string is:") print(json_string)

Output:

The JSON string is: {"employee": {"name": "John Doe", "age": "35", "job": {"title": "Software Engineer", "department": "IT", "years_of_experience": "10"}, "address": {"street": "123 Main St.", "city": "San Francisco", "state": "CA", "zip": "94102"}}} Convert XML File to JSON File in Python

We can also convert an XML file to a JSON file in python. For this, we will use the following steps.

• First, we will open the XML file in read mode using the open() function.
• Next, we will read the contents of the XML file as a string using the read() function. 
• Then, we will convert the XML string to a python dictionary using the parse() method defined in the xmltodict module.
• Now that we have the XML file as a dictionary, we will open an empty JSON file in write mode using the open() function. 
• Next, we will dump the dictionary into the json file using the dump() method defined in the JSON module. The dump() method takes the dictionary as its first argument and the file pointer as the second input argument. After execution, it saves the dictionary into the json file as a json object.
• Finally, we will close the files using the close() method.

After execution of the above steps, the data in the XML file will be saved in the JSON file. You can observe this by executing the following code.

import xmltodict import json xml_file=open("person.xml","r") xml_string=xml_file.read() python_dict=xmltodict.parse(xml_string) file=open("person.json","w") json.dump(python_dict,file) file.close()

After execution of the above code, the XML data from “person.xml” will be saved into “person.json” in JSON format.

Conclusion

In this article, we have discussed ways to convert an XML string or file into JSON format in Python. To learn more about file conversions, you can read this article on how to convert JSON to XML in python. You might also like this article on how to convert YAML to XML in Python.

I hope you enjoyed reading this article. Stay tuned for more informative articles.

Happy Learning!

The post Convert XML to JSON in Python appeared first on PythonForBeginners.com.

Watch here:

Or listen here:

Welcome back to the Pybites podcast! This week we have an inspirational talk with our special guest Cassandra Babilya! We talk about the importance of finding value and purpose in your work.

We talk about:

– Considering your life purpose.

– Your professional values and your personal values.

– How do you identify that something is wrong at work.

We discuss the effects that being disengaged with your work can have on your physical and mental well-being. The key point being to really pay attention to you “bio-feedback”:

– Are you exhausted all the time?

– Are you getting sick more often?

– Is your mental health suffering?

To get started on analysing your situation and making those first steps toward change, we ask you four questions (grab a pen and paper!):

1. Do you feel a disconnect between your personal values and your professional values?

2. The last time you jumped out of bed excited for the day, what were you going to do that day?

3. What do people often come to you for help with?

4. What’s missing when you’re not in the room?

Start small; identify where you are today and identify one thing that you can do today that will bring you closer to where you want to be.

Connect with Cassandra:
– LinkedIn
– Website
– Instagram
– Newsletter

Are you still using loops and lists to process your data in Python? Have you heard of a Python library with optimized data structures and built-in operations that can speed up your data science code? This week on the show, Jodie Burchell, developer advocate for data science at JetBrains, returns to share secrets for harnessing linear algebra and NumPy for your projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

BeautifulSoup is a popular Python library that simplifies web scraping by allowing you easily extract useful information from HTML (web) files. In this article, we will explain what User Agents are, why they are essential for web scraping, and how to use them in BeautifulSoup.

What is a User Agent?

User-Agent is a header that identifies the client making an HTTP request. It typically contains information about the client’s software, operating system, and device type. Web servers use this information to determine how to handle the request, such as serving the appropriate web page version for a mobile device.

Why is User Agents important for Web Scraping?

User-Agent is critical for web scraping because some websites block requests from known web scraping bots or scripts. Websites can identify these bots by analyzing the headers in the HTTP request. Most automated scripts and libraries like BeautifulSoup and Scrapy have a “default” user-agent, which identifies them as a bot. This makes it easy for the a website to identify and block you.

By using a User-Agent header, web scrapers can mimic a web browser and avoid detection. We can use a generated a User agent, such as the one below to fake our presence as an actual browser.

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML%2C like Gecko) Chrome/97.0.4692.99 Safari/537.36 OPR/83.0.4254.62

We will use the above user agent in this tutorial.

How to use User Agents in BeautifulSoup?

First, you need to understand that BeautifulSoup actually has nothing to do with the internet. It is not responsible for sending a request to a website and downloading the HTML content. Neither it is responsible for sending any user agents.

All of this is the job of the requests library in Python, which fetches all of the HTML content of a web page, for BeautifulSoup to analyze.

So our actual question here is: “How to use User Agents in the Requests Module“.

Let’s begin.

We will start by examining the default headers that the Requests library is sending. The user agent information is typically stored within the headers of the Request object.

import requests url = 'https://google.com' response = requests.get(url) print(response.request.headers["User-Agent"]) python-requests/2.25.1

As we can see from the output above, our python program is identifying itself as the “requests” library. This is an immediate give-away to any website that cares whether you are a bot or not.

Let’s fix this issue. By passing in a custom user agent into the headers parameters, we can over-ride the default user agent.

import requests url = 'https://google.com' headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'} response = requests.get(url, headers=headers) print(response.request.headers["User-Agent"]) Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3

As you can see form the output, this time our user agent resembles an actual browser. You are now one step closer to remaining completely undetected and safe while scraping data off the internet.

It’s important to note that the User-Agent string used in the example above is just one example of many. You can find a list of User-Agent strings for popular web browsers and devices online. It’s essential to choose a User-Agent that closely matches the web browser and device that you want to mimic to avoid detection.

This marks the end of the BeautifulSoup User Agents – Python Tutorial. Any suggestions or contributions for CodersLegacy are more than welcome. Questions regarding the article content can be asked in the comments section below.

The post BeautifulSoup User Agents in Python appeared first on CodersLegacy.

Litany against nulls.

The myths and misconceptions regarding null behavior in GraphQL are epic. Even the spec is wrong.

Inputs (such as field arguments), are always optional by default. However a non-null input type is required. In addition to not accepting the value null, it also does not accept omission. For the sake of simplicity nullable types are always optional and non-null types are always required.

One might pedantically ask, can a spec be wrong by definition? Yes, it can contradict itself.

Arguments can be required. An argument is required if the argument type is non-null and does not have a default value. Otherwise, the argument is optional.

…

Input object fields may be required. Much like a field may have required arguments, an input object may have required fields. An input field is required if it has a non-null type and does not have a default value. Otherwise, the input object field is optional.

Some of the arguments being made here could be dismissed as merely semantics. In the author’s opinion, the misinformation has clearly caused real misunderstanding. Let’s break down the persistent myths.

Fields are nullable by default.

Most type systems which recognise “null” provide both the common type and the nullable version of that type, whereby default types do not include “null” unless explicitly declared. However, in a GraphQL type system, every field is nullable by default.

This is true in the context of the spec and the reference graphql-js implementation. Whether it is relevant to a developer creating an API is entirely dependent on which graphql framework they are using.

• graphene-python: nullable by default. Types are wrapped by a NonNull type or passed a required=True flag.
• strawberry-graphql: non-null by default. Following the Python convention, types must be annotated as nullable explicitly.
• ariadne and every schema-first framework: assuming there is a default, it would be nullable.

Does the graphql schema have a “default”? In the sense that a Type must be annotated with a whole extra character to become Type!. Not in the same sense that query is the default operation, for example, because the operation can be omitted. There is no explicit Type? syntax which is being defaulted to. It is just as accurate to say there is no default in the schema; there is a syntactic binary choice.

Fields should default to nullable.

When designing a GraphQL schema, it’s important to keep in mind all the problems that could go wrong and if “null” is an appropriate value for a failed field. Typically it is, but occasionally, it’s not. In those cases, use non-null types to make that guarantee.

So not whether nulls literally are the default, but the recommendation that one should default to using nullable types. This advice would be fine in theory, but in practice has reversed what is “typical” versus “occasional”. Typically, errors propagate up to an enclosing type, possibly all the way up. It is common for trivial scalars to never be null, and like any tree there are far more leaf nodes than higher nodes.

It is also rare to want nulls in a list. Better to omit them; even more so for input lists.

The standard advice is often combined with a compatibility claim.

Including non-null fields and arguments in a schema makes that schema harder to evolve where a client expects a previously non-null field’s value to be provided in a response. For example, if a non-null email field on a User type is converted to a nullable field, will the clients that use that field be prepared to handle this potentially null value after the schema is updated? Similarly, if the schema changes in such a way that a client is suddenly expected to send a previously nullable argument with a request, then this may also result in a breaking change.

Notice the common mistake: the first sentence contradicts the last with respect to arguments. Null compatibility advice applies only to outputs; the opposite is true for inputs. Even the staunchest pro-null advocate must concede that one should “default” to inputs being non-null.

A better question to ask is “what would a null in this field represent”? In an implemented API, if the field can actually be null, then there will be an answer to that question. But if the field is truly never null, then there may be no answer. You may find yourself not just bike-shedding, but counterfactual bike-shedding. How is the client supposed to “correctly” handle a response that is impossible, in a scenario that is indescribable?

There is no need to speculate on what a hypothetical API might do, nor to agree on a “default” behavior. Analyze how the implemented API actually behaves, and describe it accurately.

Optional == Nullable.

GraphQL is far from the first to conflate “optional” with “nullable”, but it has elevated it to a next level. There is no context in which the quote “nullable types are always optional and non-null types are always required” is both true and not vacuous.

• Output field
  • From the client’s perspective, optional would mean the client can omit the field from the request. No, all fields are optional in that sense.
  • From the server’s perspective, optional would mean the server can omit the field from the response. No, all fields are required in that sense. Nullables may null, not omitted.
• Input field or argument
  • From the client’s perspective, optional would mean the client can omit the input. Yes, so they are equivalent in this case? No, an input can also be optional by having a default value, even with a non-null type. Nullable implies optional; optional does not imply nullable.
  • From the server’s perspective, optional would mean omission is equivalent to a sent null value. No, the spec clarifies that those two scenarios are semantically different.

If the value null was provided for an input object field, and the field’s type is not a non-null type, an entry in the coerced unordered map is given the value null. In other words, there is a semantic difference between the explicitly provided value null versus having not provided a value.

So there is only one context in which the concept of “optional” has any meaning: when the client can omit the input. And being nullable is only one of two ways for an input to be optional. That makes “optional” roughly 12.5% identical to “nullable”. optional == omittable != nullable

Alternatives

So rather than just be a rant, let’s use this hard-fought contrarian knowledge to create better APIs.

Default values wherever possible.

Default values are under-utilized, no surprise given the introduction.

length(unit: LengthUnit = METER): Float

Arguments can be either required or optional. When an argument is optional, we can define a default value - if the unit argument is not passed, it will be set to METER by default.

Notice the misleading phrasing combined with the nullable LengthUnit reinforces the false - and widespread - notion that the argument must be nullable to be optional. It would be perfectly valid for a client to send an explicit null in this example, and a good chance it is broken on the server, assuming “working” was defined. A non-null here is better in every measurable way: length(unit: LengthUnit! = METER): Float.

• the client does not have to guess what sending an explicit null would mean because it is disallowed
• the server does not have to document or support a use case that never really existed

It can not be overstated how little known default value optionality is. Once pointed out, it is impossible to not see this in APIs everywhere.

• the vast majority of Boolean inputs should be Boolean! = {false,true}
• the vast majority of [Type!] inputs should be [Type!]! = []. Any valid value is a valid default value, not just scalars.
• many Int and Float inputs should be Int! = 0 and Float! = 0.0
• some String inputs should be String! = ""

A similar dynamic has occurred in Python. None is over-used as a default value, instead of a natural default. Speaking of which, Python also fell for Optional equivocation, but at least is trying to walk it back. ... | None is the new preferred syntax, and the documentation is quick to point out that the Optional annotation does not actually mean optional.

Exploit the explicit null distinction.

There are at least two scenarios where distinguishing an explicit null from omission is quite useful.

A partial update where null is a valid value to set, that is an unset. Often there will be a clunky Boolean flag to indicate “no seriously, set the null”: update(name: String, setName: Boolean, ...). Instead, drop the flag and document that passing a value will update (null or otherwise), and omission will not update.

Another scenario is a filter where null is a valid value. It is a perfectly natural interpretation that omission means to not apply the filter, whereas an explicit null means to filter on null.

If this path is followed, it also becomes natural to add = null to inputs where there is no difference. Yes, even null is a valid default value. The point is it clearly documents to the client that there is no difference, while providing assurance that the server is implemented correctly.

Note many frameworks will not make it convenient to check whether an input was present. It may be necessary to check the argument map in the GraphQL info. One of the few that does is strawberry-graphql, which uses UNSET as a sentinel. Using a sentinel when None is a valid value is an established pattern in Python, such as dataclasses.MISSING.

Use an @optional directive.

There is one last case that is not well-covered. What if there is not a natural default, and no inherent meaning for null. That is, the server is being forced to declare an input as nullable when it only wants it to be optional. Using an @optional directive would clarify that, and remove any expectation of behavior if the client insists on sending a null.

Advocacy

The above suggestions are in this GraphQL proposal, and can be see implemented in the graphique project. It demonstrates fields like: slice(offset: Int! = 0, length: Int = null, reverse: Boolean! = false).

When debating this topic, beware of undue JavaScript influence from a client perspective. For example, undefined is neither a GraphQL nor a JSON concept. And all of the input points listed apply to input coercion of variables as well.

Introduction

Multiline strings are a useful feature in Python that allow you to create strings that span multiple lines of code. They can be especially helpful when you need to create long or complex strings, such as for formatting text, writing documentation, or storing data.

In this article, we'll explore three different methods for creating multiline strings in Python. First, we'll look at the most common method - using triple quotes. Then, we'll cover a second method that uses escape characters to continue a string onto a new line. Finally, we'll explore a third method that uses the join() method to concatenate a list of strings into a single multiline string.

By the end of this article, you'll have a clear understanding of the different ways you can create multiline strings in Python, as well as the advantages and limitations of each method. This knowledge will help you choose the best method for your specific use case, and make your Python code more readable and efficient. So, let's get started!

Method 1: Using Triple Quotes

The most common way to create a multiline string in Python is to use triple quotes. Triple quotes are a set of three quotation marks that allow you to create a string that spans multiple lines.

Note: In this use case, there is no difference between single (') and double (") quotation marks - so, you can use either triple single quotation marks ('''), as well as triple double quotation marks (""").

Let's take a look at an example of using triple double quotes to create a multiline string:

multiline_string = """ This is a multiline string that spans multiple lines. We can use it for long strings that need to be formatted or displayed across several lines. """

The string starts and ends with three quotation marks, which tells Python that it's a multiline string. The text between the quotes can span as many lines as needed, and any line breaks or white space will be preserved in the resulting string.

One of the main advantages of using triple quotes to create multiline strings is that they are easy to read and maintain. You can see exactly how the string is formatted and organized, without needing to use escape characters or other tricks to continue the string on multiple lines.

However, one limitation of this method is that it can be difficult to use triple quotes within the string itself, since this can cause conflicts with the opening and closing quotes. In such cases, you might need to use another method, such as escape characters or the join() method, to create your multiline string.

Overall, using triple quotes is a straightforward and effective way to create multiline strings in Python.

Method 2: Using Escape Characters

Another way to create a multiline string in Python is to use escape characters to continue the string on a new line. Escape characters are special characters that start with a backslash (\) and indicate that the next character should be treated in a special way.

Here's an example of using escape characters to create a multiline string:

multiline_string = "This is a multiline string \ that uses escape characters \ to continue the string on a new line."

Here, we effectively use the backslash character (\) in front of the new line character to indicate that the string should continue on the next line. The backslash itself is not included in the resulting string, so the string appears as a single line of text. It is simply there to help you create a long string on multiple lines.

One advantage of using escape characters to create multiline strings is that it allows you to use other special characters within the string, such as quotes or newlines, without causing conflicts with the opening and closing quotes. However, this method can be more difficult to read and maintain than using triple quotes, especially for longer strings that span many lines.

Using escape characters to create a multiline string can be a useful alternative to using triple quotes, especially if you need to use special characters within the string. However, be sure to only use this method when it makes sense and that the resulting string is still easy to read and understand. We want to improve the readability of our code, not worsen it!

Method 3: Using the join() Method

A third way to create a multiline string in Python is to use the join() method to concatenate a list of strings into a single string. This method can be especially useful if you already have a list of strings that you want to combine into a multiline string.

Let's show how to use the join() method to create a multiline string:

lines = [ "This is the first line", "This is the second line", "This is the third line" ] multiline_string = "\n".join(lines)

In this example, we first create a list of strings, with each string representing a new line of text. We then use the join() method to concatenate the strings into a single string, with each string separated by a newline character (\n). The resulting string is a multiline string that contains all of the lines from the original list.

Using the join() method to create a multiline string is that it can be a more efficient method for joining long strings, since it avoids creating intermediate strings at each step. Additionally, this method can be useful if you already have a list of strings that you want to combine into a multiline string.

Note: You can use any character or string to join the lines together. So while we use a newline character in our example here, you could theoretically use a space or many other options.

However, one limitation of this method is that it requires you to first create a list of strings, which can be more cumbersome than simply typing out the multiline string directly. Additionally, this method may not be as readable or easy to maintain as using triple quotes, especially for longer strings that span many lines.

Overall, using the join() method to create a multiline string can be a useful alternative to using triple quotes or escape characters, especially if you already have a list of strings that you want to combine into a single string.

Which Method to Choose

Here we've seen three different methods for creating multiline strings in Python: using triple quotes, using escape characters, and using the join() method. Each of these methods has its own advantages and limitations, and the best method to use will depend on the specific use-case.

Let's take a look at a brief comparison of the three methods:

• Triple Quotes
  • Pros: Easy to read and maintain, and can include triple quotes within the string
  • Cons: Limited if you need to use triple quotes within the string itself
• Escape Characters
  • Pros: Useful for creating multiline strings with special characters
  • Cons: More difficult to read and maintain than triple quotes, and can become cumbersome for longer strings
• join() Method
  • Pros: Useful if you already have a list of strings that you want to combine into a multiline string, and can be a more efficient method for joining long strings
  • Cons: Requires you to first create a list of strings, and may not be as readable as other methods

Overall, the method you choose to create a multiline string in Python will depend on your specific use case and the preferences of you and your team. In general, it's a good idea to choose the method that is easiest to read and maintain, while still meeting your functional requirements.

Conclusion

Creating multiline strings is a common task in Python, and there are a number of ways to achieve it. In this article, we explored three different methods: using triple quotes, using escape characters, and using the join() method.

Each of these methods has its own advantages and limitations, and the best method to use will depend on your specific use case. Overall, the most commonly used method is the triple quotes, but using escape characters or the join() method may be more appropriate, depending on your situation.

We hope this article has been helpful in explaining the different methods for creating multiline strings in Python. Whether you're working on a personal project or a large-scale software application, understanding these methods will help you write efficient and, more importantly, readable code.

Fundamentals of C programming continued...

<p>How do we check if something is <code>None</code>?</p> <p>With the beauty of the Python language - the code that you would write is literally the same as the above question:</p> <pre class="language-python"><code class="language-python"><span class="token keyword">if</span> something <span class="token keyword">is</span> <span class="token boolean">None</span><span class="token punctuation">:</span></code></pre> <p>It reminds me of this joke:</p> <blockquote> <p><em>- How do you turn pseudocode into Python?</em><br /> <em>- You add <code>.py</code> at the end of the file.</em></p> </blockquote> <p>There is another way in which we could make this comparison:</p> <pre class="language-python"><code class="language-python"><span class="token keyword">if</span> something <span class="token operator">==</span> <span class="token boolean">None</span><span class="token punctuation">:</span></code></pre> <p>However, it doesn't make sense to use the second variant. <code>None</code> <a href="https://stackoverflow.com/questions/38288926/in-python-is-none-a-unique-object">is a singleton object</a> - there can't be two different <code>None</code> objects in your code. Each time you assign <code>None</code> to a variable, you reference the same <code>None</code>:</p> <pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> a <span class="token operator">=</span> <span class="token boolean">None</span><br /><span class="token operator">>></span><span class="token operator">></span> b <span class="token operator">=</span> <span class="token boolean">None</span><br /><span class="token operator">>></span><span class="token operator">></span> c <span class="token operator">=</span> <span class="token boolean">None</span><br /><span class="token operator">>></span><span class="token operator">></span> a <span class="token keyword">is</span> b <span class="token keyword">is</span> c<br /><span class="token boolean">True</span></code></pre> <p>To compare the identity, you should use <code>is</code>, rather than <code>==</code>, as I explained in the <a href="https://switowski.com/blog/checking-for-true-or-false/">Checking for True or False</a> article. It's clearer and faster:</p> <pre class="language-shell"><code class="language-shell">$ python -m timeit -s <span class="token string">"a = 1"</span> <span class="token string">"a is None"</span><br /><span class="token number">50000000</span> loops, best of <span class="token number">5</span>: <span class="token number">8.2</span> nsec per loop<br /><br />$ python -m timeit -s <span class="token string">"a = 1"</span> <span class="token string">"a == None"</span><br /><span class="token number">20000000</span> loops, best of <span class="token number">5</span>: <span class="token number">13</span> nsec per loop</code></pre> <p>As you can see, <code>==</code> is 60% slower than <code>is</code> (13 / 8.2 ≈ 1.59).</p>

In today’s video, you will learn about the following topics:

• An intro to Test Driven Development (TDD)
• Red-Green-Refactor
• The importance of source control
• Code coverage in Python

I’ll show you how I use pytest to test your code and Coverage to learn how much of your code is truly tested. You will use the FizzBuzz Code Kata to learn about all these topics.

The post An Intro to Testing in Python with a Code Kata (Video) appeared first on Mouse Vs Python.

The terminal can be intimidating to work with when you’re used to working with graphical user interfaces (GUIs). However, it’s an important tool that you need to get used to in your journey as a Python developer. And once you level up your skill of using the terminal, it becomes an extremely powerful tool in your repertoire. With just a few commands in the terminal, you can do tasks that are impossible or at least very tedious to do in a GUI.

In this tutorial, you’ll learn how to:

• Find the terminal on your operating system
• Open the terminal for the first time
• Navigate your file system with basic commans
• Create files and folders with the terminal
• Manage packages with pip commands
• Keep track of your files with Git in the terminal

If you’re new to working with the terminal, or you’re looking to expand your understanding of its capabilities, then this tutorial is a great starting point. In it, you’ll get an introduction to some of the basic commands and learn how to use pip and Git to manage your projects in the terminal.

Understanding how to integrate the terminal, pip, and Git into your workflows is essential for you as a Python developer. However, it’s important to note that you’ll only scratch the surface of what the terminal can do, and there’s much more to learn as you continue to explore the terminal as an essential development tool.

Free Download: Click here to get a free cheat sheet of useful commands to get you started working with the terminal.

Install and Open the Terminal

Back in the day, the term terminal referred to some clunky hardware that you used to enter data into a computer. Nowadays, people are usually talking about a terminal emulator when they say terminal, and they mean some kind of terminal software that you can find on most modern computers.

Note: There are two other terms that you might hear now and then in combination with the terminal:

1. A shell is the program that you interact with when running commands in a terminal.
2. A command-line interface (CLI) is a program designed to run in a shell inside the terminal.

In other words, the shell provides the commands that you use in a command-line interface, and the terminal is the application that you run to access the shell.

If you’re using a Linux or macOS machine, then the terminal is already built in. You can start using it right away.

On Windows, you also have access to command-line applications like the Command Prompt. However, for this tutorial and terminal work in general, you should use the Windows terminal application instead.

Read on to learn how to install and open the terminal on Windows and how to find the terminal on Linux and macOS.

Windows

The Windows terminal is a modern and feature-rich application that gives you access to the command line, multiple shells, and advanced customization options. If you have Windows 11 or above, chances are that the Windows terminal is already present on your machine. Otherwise, you can download the application from the Microsoft Store or from the official GitHub repository.

Before continuing with this tutorial, you need to get the terminal working on your Windows computer. You can follow the Your Python Coding Environment on Windows: Setup Guide to learn how to install the Windows terminal.

After you install the Windows terminal, you can find it in the Start menu under Terminal. When you start the application, you should see a window that looks like this:

It can be handy to create a desktop shortcut for the terminal or pin the application to your task bar for easier access.

Linux

You can find the terminal application in the application menu of your Linux distribution. Alternatively, you can press Ctrl+Alt+T on your keyboard or use the application launcher and search for the word Terminal.

After opening the terminal, you should see a window similar to the screenshot below:

How you open the terminal may also depend on which Linux distribution you’re using. Each one has a different way of doing it. If you have trouble opening the terminal on Linux, then the Real Python community will help you out in the comments below.

macOS

A common way to open the terminal application on macOS is by opening the Spotlight Search and searching for Terminal. You can also find the terminal app in the application folder inside Finder.

Read the full article at https://realpython.com/terminal-commands/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

XML and YAML are two of the most used file formats in software engineering. Sometimes, we need to convert an XML file to YAML or vice versa. This article discusses how to convert XML string or file to YAML format in Python.

Table of Contents

1. What is XML Format?
2. What is the YAML File Format?
3. Convert XML String to YAML String in Python
4. XML String to YAML File in Python
5. Convert XML File to YAML String in Python
6. Convert XML File to YAML File in Python
7. Conclusion

What is XML Format?

XML is a markup language used for encoding documents in a human and machine-readable format. It defines elements using tags, which enclose content and can have attributes for additional information. XML is commonly used for data exchange and defining data formats between different systems. It can be verbose and complex compared to other formats, such as JSON or YAML.

Following is an XML document that contains the data of an employee.

<?xml version="1.0"?> <employee> <name>John Doe</name> <age>35</age> <job> <title>Software Engineer</title> <department>IT</department> <years_of_experience>10</years_of_experience> </job> <address> <street>123 Main St.</street> <city>San Francisco</city> <state>CA</state> <zip>94102</zip> </address> </employee>

This XML string represents an employee record. It starts with an XML declaration specifying the version of the standard XML.

• The root element is <employee>, which contains child elements for the employee’s name, age, job, and address.
• The employee’s name is contained in the <name> element, and their age is contained in the <age> element.
• The <job> element contains child elements for the employee’s job title, department, and years of experience.
• The <address> element contains child elements for the employee’s street address, city, state, and zip code.

The XML format is widely used in different areas, including web services, document processing, and data exchange between different systems.

One advantage of XML is that it allows for the definition of structured data formats. This can be helpful in ensuring that data is consistent and accurate. However, because XML can be verbose and more complex than other formats such as JSON or YAML, it may not always be the best choice for all situations.

What is the YAML File Format?

YAML (short for “YAML Ain’t Markup Language”) is a human-readable data serialization format often used for configuration files, data exchange between different systems, and storing structured data. It is designed to be easy to read and write for both humans and machines and uses indentation and simple syntax to define data structures.

YAML supports a wide range of data types, including strings, numbers, booleans, arrays, and maps (also known as dictionaries or hashes). It allows for comments and references between different document parts and can include complex data structures such as nested arrays and maps.

The data shown in the previous XML example can be stored in the YAML file format as shown below.

employee: name: John Doe age: 35 job: title: Software Engineer department: IT years_of_experience: 10 address: street: 123 Main St. city: San Francisco state: CA zip: 94102

Look at the increased readability of the data. That’s why we use YAML files more for storing configurations and structured data.

Convert XML String to YAML String in Python

To convert an XML string to YAML, we will use the xmltodict module and the yaml module. For this, we will use the following steps.

First, we will convert the XML string to a python dictionary using the parse() method defined in the xmltodict module. The parse() method takes an XML string as its input argument and returns a python dictionary after execution.

After this, we will convert the python dictionary to a YAML string using the dump() method defined in the yaml module. The dump() method takes a dictionary as its input argument and returns a YAML string.

You can observe this in the following example.

import xmltodict import yaml xml_string="""<?xml version="1.0"?> <employee> <name>John Doe</name> <age>35</age> <job> <title>Software Engineer</title> <department>IT</department> <years_of_experience>10</years_of_experience> </job> <address> <street>123 Main St.</street> <city>San Francisco</city> <state>CA</state> <zip>94102</zip> </address> </employee>""" print("The XML string is:") print(xml_string) python_dict=xmltodict.parse(xml_string) yaml_string=yaml.dump(python_dict) print("The YAML string is:") print(yaml_string)

Output:

The XML string is: <?xml version="1.0"?> <employee> <name>John Doe</name> <age>35</age> <job> <title>Software Engineer</title> <department>IT</department> <years_of_experience>10</years_of_experience> </job> <address> <street>123 Main St.</street> <city>San Francisco</city> <state>CA</state> <zip>94102</zip> </address> </employee> The YAML string is: employee: address: city: San Francisco state: CA street: 123 Main St. zip: '94102' age: '35' job: department: IT title: Software Engineer years_of_experience: '10' name: John Doe XML String to YAML File in Python

You can convert an XML string to a YAML file instead of a string. To convert an XML string to a YAML file, you can use the following steps.

• First, we will convert the XML string to a python dictionary using the parse() method defined in the xmltodict module.
• Next, we will open an empty YAML file using the open() function. The open() function takes the file name as its first input argument and the python literal “w” as its second input argument. After execution, it returns a file pointer.
• Now, we will dump the python dictionary to the YAML file using the dump() method defined in the yaml module. The dump() method takes the dictionary as its first argument and the file pointer as its second input argument.
• Finally, we will close the file using the close() method. After this, the yaml file will be saved to the storage.

You can use the above steps to convert an XML string to a YAML file as shown below.

import xmltodict import yaml xml_string="""<?xml version="1.0"?> <employee> <name>John Doe</name> <age>35</age> <job> <title>Software Engineer</title> <department>IT</department> <years_of_experience>10</years_of_experience> </job> <address> <street>123 Main St.</street> <city>San Francisco</city> <state>CA</state> <zip>94102</zip> </address> </employee>""" python_dict=xmltodict.parse(xml_string) file=open("person.yaml","w") yaml.dump(python_dict,file) file.close()

The output file looks as follows.

YAML File Convert XML File to YAML String in Python

We can also convert an XML file to a YAML string. For this, we can use the following steps.

• First, we will open the XML file in read mode using the open() function. Here, the open() function takes the file name as its first input argument and the literal “r” as its second input argument. After execution, it returns a file pointer.
• Next, we will read the contents of the XML file using the read() method. The read() method, when invoked on the file pointer, returns the file contents as a string.
• Once we get the XML string, we will convert it to a python dictionary using the parse() method defined in the xmltodict module. The parse() method takes the XML string as its input argument and returns the python dictionary.
• Finally, we will use the dump() method defined in the yaml module to convert the python dictionary to yaml string. The dump() method takes the dictionary as its input argument and returns the yaml string.

We will convert the following XML file to YAML.

XML File

 

You can observe the entire process in the following example.

import xmltodict import yaml xml_file=open("person.xml","r") xml_string=xml_file.read() python_dict=xmltodict.parse(xml_string) yaml_string=yaml.dump(python_dict) print("The YAML string is:") print(yaml_string)

Output:

The YAML string is: employee: address: city: San Francisco state: CA street: 123 Main St. zip: '94102' age: '35' job: department: IT title: Software Engineer years_of_experience: '10' name: John Doe Convert XML File to YAML File in Python

To convert an XML file to a YAML file in python, you first need to open the XML file and read it into a python dictionary. Then, you can open a yaml file and dump the dictionary into the yaml file as shown below.

import xmltodict import yaml xml_file=open("person.xml","r") xml_string=xml_file.read() python_dict=xmltodict.parse(xml_string) file=open("person.yaml","w") yaml.dump(python_dict,file) file.close() Conclusion

In this article, we have discussed how to convert an XML string to YAML in Python. To learn more about file conversions, you can read this article on how to convert YAML to XML in Python. You might also like this article on how to convert a Python dictionary to XML.

I hope you enjoyed reading this article. Stay tuned for more informative articles.

Happy Learning!

The post Convert XML to YAML in Python appeared first on PythonForBeginners.com.

fundamentals of C programming language series continued...

So you're trying to read some YAML using PyYAML, and get an exception like this:

>>> yaml.safe_load("""\ ... [0, 0]: top-left ... [1, 1]: bottom-right ... """) Traceback (most recent call last): ... yaml.constructor.ConstructorError: while constructing a mapping found unhashable key in "<unicode string>", line 1, column 1: [0, 0]: top-left ^ What does it mean? #

The error message is pretty self-explanatory, but let's unpack it a bit.

First, it happened during construction – that is, while converting the generic representation of the YAML document to native data structures; in this case, converting a mapping to a Python dict.

YAML Processing Overview

The problem is that a key of the mapping, [0, 0], is not hashable:

An object is hashable if it has a hash value which never changes during its lifetime (it needs a __hash__() method), and can be compared to other objects (it needs an __eq__() method). Hashable objects which compare equal must have the same hash value.

Most immutable built-ins are hashable; mutable containers (such as lists) are not; immutable containers (such as tuples) are hashable only if their elements are.

Why does this happen? #

This is not a limitation of YAML itself; quoting the spec:

The content of a mapping node is an unordered set of key/value node pairs, with the restriction that each of the keys is unique. YAML places no further restrictions on the nodes. In particular, keys may be arbitrary nodes, the same node may be used as the value of several key/value pairs and a mapping could even contain itself as a key or a value.

We can load everything up to the representation:

>>> yaml.compose("[0, 0]: top-left") MappingNode( tag='tag:yaml.org,2002:map', value=[ ( SequenceNode( tag='tag:yaml.org,2002:seq', value=[ ScalarNode(tag='tag:yaml.org,2002:int', value='0'), ScalarNode(tag='tag:yaml.org,2002:int', value='0'), ], ), ScalarNode(tag='tag:yaml.org,2002:str', value='top-left'), ) ], )

The limitation comes from how dicts are implemented, specifically:

Hashability makes an object usable as a dictionary key and a set member, because these data structures use the hash value internally.

>>> {[0, 0]: "top-left"} Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unhashable type: 'list'

If we use a (hashable) tuple instead, it works:

>>> {(0, 0): "top-left"} {(0, 0): 'top-left'} What now? # Use the representation #

Depending on your needs, the representation might be enough. But probably not...

Use full_load() and Python-specific tags #

If you control the input, and you're OK with language-specific tags, use full_load(); it resolves all tags except those known to be unsafe, including all the Python-specific tags listed here.

>>> yaml.full_load("""\ ... !!python/tuple [0, 0]: top-left ... !!python/tuple [1, 1]: bottom-right ... """) {(0, 0): 'top-left', (1, 1): 'bottom-right'}

You could also use unsafe_load(), but most of the time it's not what you want:

Warning

yaml.unsafe_load() is unsafe for untrusted data, because it allows running arbitrary code. Consider using safe_load() or full_load() instead.

For example, you can do this:

>>> yaml.unsafe_load("!!python/object/new:os.system [echo WOOSH. YOU HAVE been compromised]") WOOSH. YOU HAVE been compromised 0

There were a bunch of CVEs about it.

But, I don't control the input #

If you don't control the input, you can use a custom constructor to convert the keys to something hashable. Here, we convert list keys to tuples:

class Loader(yaml.SafeLoader): pass def construct_mapping(self, node): pairs = self.construct_pairs(node, deep=True) try: return dict(pairs) except TypeError: rv = {} for key, value in pairs: if isinstance(key, list): key = tuple(key) rv[key] = value return rv Loader.construct_mapping = construct_mapping Loader.add_constructor('tag:yaml.org,2002:map', Loader.construct_mapping) >>> yaml.load("""\ ... [0, 0]: top-left ... [1, 1]: bottom-right ... """, Loader=Loader) {(0, 0): 'top-left', (1, 1): 'bottom-right'}

We subclass SafeLoader to account for a PyYAML quirk – calling add_constructor() directly would modify it in-place, for everyone, which isn't necessarily great. We still override construct_mapping so that other constructors wanting to make a mapping get to use our version.

Alas, this is quite limited, because new key types need to be handled explicitly; for example, we might be able to convert dicts to a frozenset of items().

>>> yaml.load("{0: 1}: top-left", Loader=Loader) Traceback (most recent call last): ... rv[key] = value TypeError: unhashable type: 'dict'

But nested keys don't work either, we need to convert them recursively ourselves:

>>> yaml.load("[[0]]: top-left", Loader=Loader) Traceback (most recent call last): ... rv[key] = value TypeError: unhashable type: 'list' But, I don't know the key types in advance #

A decent trade-off is to just let the mapping devolve into a list of pairs:

def construct_mapping(self, node): pairs = self.construct_pairs(node) try: return dict(pairs) except TypeError: return pairs >>> yaml.load("""\ ... [0, 0]: top-left ... [1, 1]: bottom-right ... """, Loader=Loader) [([0, 0], 'top-left'), ([1, 1], 'bottom-right')] >>> yaml.load("{0: 1}: top-left", Loader=Loader) [({0: 1}, 'top-left')] But, I need to round-trip the data #

This works, until you need to round-trip the data, or emit this kind of YAML yourself.

Luckily, I've already written a whole article on how to do that, complete with code; the trick is to mark the list of pairs in some way:

>>> value = yaml.load("""\ ... [0, 0]: top-left ... [1, 1]: bottom-right ... """, Loader=Loader) >>> value Pairs([([0, 0], 'top-left'), ([1, 1], 'bottom-right')])

... so you can represent it back into a mapping:

>>> print(yaml.dump(value, Dumper=Dumper)) ? - 0 - 0 : top-left ? - 1 - 1 : bottom-right

(The fancy ?  syntax indicates a complex mapping key, but that's just another way of writing the original input.)

That's it for now.

Learned something new today? Share this with others, it really helps! Twitter HN Reddit

Want to know when new articles come out? Subscribe here to get new stuff straight to your inbox!

#565 – FEBRUARY 21, 2023
View in Browser »

How to Flush the Output of the Python Print Function

In this tutorial, you’ll learn how to flush the output of Python’s print function. You’ll explore output stream buffering in Python using code examples and learn that output streams are block-buffered by default, and that print() with its default arguments executes line-buffered when interactive.
REAL PYTHON

The Heisenbug Lurking in Your Async Code

When using the create_task() function in asyncio it is very important to maintain a reference to the created tasks. Although this requirement is documented, it is easy to forget and can have some very hard to understand consequences.
WILL MCGUGAN

Leaving Serverless Postgres for Crunchy Bridge

Find out why vanilla Postgres is a more cost effective and simple way to manage your database than Postgres “compatible” databases. Read a customer case study on lowering costs, getting a better database experience, and better support on Crunchy Bridge →
CRUNCHY DATA sponsor

What’s New in SQLAlchemy 2.0?

SQLAlchemy 2.0 was launched in January. This article reviews the latest changes, whether it is worth the upgrade, and how to go about it.
MIGUEL GRINBERG

PyCon US 2023 (Salt Lake City, Utah) Schedule Available

PYCON.BLOGSPOT.COM

Django Security Releases Issued: 4.1.7, 4.0.10, and 3.2.18

DJANGO SOFTWARE FOUNDATION

Discussions Why Type Hinting Sucks!

REDDIT

Python Jobs Software Engineer - Backend/Python (100% Remote) (Anywhere)

Close

Python Video Course Instructor (Anywhere)

Real Python

Python Tutorial Writer (Anywhere)

Real Python

More Python Jobs >>>

Articles & Tutorials Latest Attack on PyPI Users, Crooks Are Getting Better

Over 400 new malicious packages have been uploaded to PyPI that use a malicious JavaScript extension to monitor infected machines for crypto-currency interactions with the intent of stealing credentials. Packages are named based on typos of many of the most popular PyPI downloads.
DAN GOODIN

PySport Open Source

This site is a collection of open source Python and R tools for sports analytics, including scrapers and API wrappers for a variety for sites, data plotting, and analysis. It is maintained by PySport.
PYSPORT.ORG

Show Open-Source Some Love by Joining The Big Fix 2023

Join Snyk between February 14 and March 14 to help make security x100 times better in 2023 with The Big Fix event! Find and fix vulnerabilities while networking with your fellow engineers. Register for free →
SNYK.IO sponsor

Getters and Setters in Python

In this video course, you’ll learn what getter and setter methods are, how Python properties are preferred over getters and setters when dealing with attribute access and mutation, and when to use getter and setter methods instead of properties in Python.
REAL PYTHON course

Python Basics: Building Systems With Classes

In this video course, you’ll learn how to work with classes to build complex systems in Python. By composing classes, inheriting from other classes, and overriding class behavior, you’ll harness the power of object-oriented programming (OOP).
REAL PYTHON course

The Essential Django Deployment Guide

Going from “it works on my machine” to deploying to the public can be a daunting task. This guide details the choices between VPS and PaaS systems, how to choose, what the options are, and what you need to know to get your Django code live.
SAAS PEGASUS

The Dangers Behind Image Resizing

When training an ML model on image data you likely want smaller, consistently sized images. That means image processing in your pipeline, but the expectation that image resizing is the same across libraries can cause unforeseen problems.
ZURU.TECH

Should You Write a class Without @dataclass?

This article makes the argument that you should always use @dataclass for building classes in Python. Read on to understand why, and maybe respond to Glyph’s call-to-action to tell him if you think he’s wrong.
GLYPH LEFKOWITZ

Caching a Lot of Methods in Python

This article goes deep on functools.lru_cache() including all the arguments against using it, and how to argue with the arguments. Learn how to cache the results of your functions with a single line of code.
ADRIAN

Google OSS Project Python Style Guide

This is the style guide for any Google-originated open-source projects and a conversation on Hacker News about its content.
GOOGLE

Projects & Code ApiLogicServer: Customizable Database Web App Projects

GITHUB.COM/VALHUBER

Pathfinding-Visualizer: Visualize Pathfinding With Pygame

GITHUB.COM/TAUSEEF-HILAL

django-prose: Wonderful Rich-Text Editing for Django

GITHUB.COM/WITHLOGICCO

CausalPy: Causal Inference in Quasi-Experimental Settings

GITHUB.COM/PYMC-LABS

Events PyCon Namibia 2023

February 21 to February 24, 2023
PYCON.ORG

PyKla Monthly Meetup

February 22, 2023
MEETUP.COM

Heidelberg Python Meetup

February 22, 2023
MEETUP.COM

Weekly Real Python Office Hours Q&A (Virtual)

February 22, 2023
REALPYTHON.COM

SPb Python Drinkup

February 23, 2023
MEETUP.COM

PyLadies Amsterdam + MLOps.Community Meetup

February 23, 2023
MEETUP.COM

PyCon PH 2023

February 25 to February 27, 2023
PYTHON.PH

PyDelhi User Group Meetup

February 25, 2023
MEETUP.COM

Happy Pythoning!
This was PyCoder’s Weekly Issue #565.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

Introduction

In the world of software development, exchanging data between different systems is a common task. One popular format for data exchange is JSON (JavaScript Object Notation), which is a lightweight and easy-to-read format for data representation. In Python, JSON data can be easily converted to dictionary objects, and vice versa. In this article, we will explore the process of converting between JSON and dictionaries in Python, and provide examples of how to do it efficiently.

Dictionaries

In Python, a dictionary is a mutable data structure that stores key-value pairs. Each key in a dictionary maps to a specific value, and the keys must be unique. Dictionaries are a very convenient way to store and retrieve data, as they allow fast lookups and support a variety of operations such as adding, updating and deleting elements.

To create a dictionary in Python, we enclose the key-value pairs in curly braces and separate them with commas. For example, the following code creates a dictionary that contains a list of people:

dict_example = {'people':[{'name': 'Cassia', 'website': 'stackabuse.com', 'country': 'Brazil'}]}

To determine the type of a variable in Python, we can use the type() function. For example, the following code will print the type of the dict_example variable:

print(type(dict_example))

This will output <class 'dict'>, which tells us that dict_example is a dictionary.

JSON and dictionaries are very similar in structure, as both use key-value pairs to represent data. The main difference is that JSON is a text-based format that is mostly used for exchanging data between different systems, whether it's via a RESTful API or saving data to files. Dictionaries, on the other hand, are a data type in Python that are used for storing and manipulating data within your code. Regardless of their differences, since JSON and Python dictionaries share a similar structure, it's easy to convert between them using Python's built-in functions.

JSON

JSON is a popular format for exchanging data. It's meant to be human-readable, and it consists of a collection of key-value pairs, similar to Python dictionaries. In Python, we can use the json library to work with JSON data, which we'll see in this article.

Raw JSON data is simply a string, which has a consistent format that can be parsed by any language. For example, here is a raw JSON string that represents information about myself:

'{"name": "Cassia", "website": "stackabuse.com", "country": "Brazil"}'

While JSON and Python dictionaries do have a similar structure, there are some differences between the two. For instance, in JSON, all keys must be strings enclosed in double quotes, whereas in Python dictionaries, keys can be any immutable type. Additionally, JSON only allows for the use of simple data types, such as strings, numbers, and booleans, whereas Python dictionaries can contain any valid Python data type.

Converting JSON to Dictionaries

Converting JSON to a Python dictionary is a common task, particularly when working with external APIs or data sources. By converting JSON data to a Python dictionary, we can manipulate the data using Python's built-in libraries, making it easier to work with. Otherwise you'd just have a raw string, which is very difficult to read and modify, even if it's structured. Hence the need for JSON parsers.

To convert the data, we can use the json.loads() method to convert a raw JSON string to a Python dictionary. Here is a simple example using the same data as above:

import json json_string = '{"name": "Cassia", "website": "stackabuse.com", "country": "Brazil"}' dictionary = json.loads(json_string) print(dictionary)

Output:

{'name': 'Cassia', 'website': 'stackabuse.com', 'country': 'Brazil'}

In this example, we first import the json library. We then create a raw JSON string and use the json.loads() method to convert it to a dictionary. Finally, we print the resulting dictionary.

It's important to note that when working with JSON data, you should almost always have error handling in place in case the JSON isn't properly formatted. Otherwise the json.loads() method will raise a JSONDecodeError exception.

Here is another example, where we load JSON data from a file:

import json with open('data.json', 'r') as f: json_data = f.read() dictionary = json.loads(json_data) print(dictionary)

In this example, we use Python's built-in open() method to read JSON data from a file. We then use the json.loads() method to convert the JSON data to a dictionary, which we then print to the console to confirm.

Converting Dictionaries to JSON

As you may have guessed, converting a Python dictionary to JSON is also a common task in software development, particularly when working with data that needs to be sent to external systems or APIs. By converting a dictionary to JSON, we can easily serialize the data and transmit it across different platforms. This is done when the data needs to be transmitted and not manipulated, like we would do with a dictionary.

To do this in Python, we can use the json.dumps() method to convert a dictionary to JSON format:

import json dictionary = {'name': 'Cassia', 'website': 'stackabuse.com', 'country': 'Brazil'} json_data = json.dumps(dictionary) print(json_data)

Output:

{"name": "Cassia", "website": "stackabuse.com", "country": "Brazil"}

In this example, we first import the json library. We create a Python dictionary for the purpose of our example and use the json.dumps() method to convert it to a JSON string. Finally, we print the resulting JSON data to view the output of dumps(). You may notice that it doesn't look all that different, but in this format it's a raw string instead of a Python object.

It's important to note that the resulting JSON data may not preserve the original order of keys in the dictionary. However, this is not typically an issue, as JSON data is typically accessed using keys, rather than by position.

Here is another example of a common use-case, where we save a dictionary to a file in JSON format:

import json dictionary = {'name': 'Cassia', 'website': 'stackabuse.com', 'country': 'Brazil'} with open('data.json', 'w') as f: json.dump(dictionary, f)

In this example, we use Python's built-in open() method to write a dictionary to a file in JSON format. We use the json.dump() method to convert the dictionary to JSON and write it to the file.

Conclusion

In conclusion, converting JSON to dictionaries in Python is a common operation that developers need to perform when working with JSON data, and vice versa. Python provides built-in support for both encoding Python data structures to JSON and also for decoding JSON data into Python objects, making it easy to convert between the two formats.

In this article, we've covered the basics of converting JSON to dictionaries in Python, and vice versa. We've explored how to use Python's built-in json module. There are also several third-party libraries available that provide more advanced features and customization options for handling JSON data in Python that we did not cover, such as simplejson and ujson.

It's important to remember that when converting between JSON and dictionaries, it's essential to ensure that the data is formatted correctly and that the data types match on both sides. By following the best practices and using the right tools, developers can effectively work with JSON data in Python, allowing them to build robust and efficient web applications.

It’s only a few months until the next PyLadies Auction at PyCon 2023!

2022's auctioneers

If you have yet to attend a PyLadies Auction at previous PyCons, don’t miss out this year. Help us raise money for PyLadies, an international mentorship group that focuses on helping more women become active participants and leads within the Python open source community.

Not only will your donations make a huge impact, but you will also have a great evening with lots of laughter, friendly competition, and unique, one-of-a-kind items. 

The PyLadies’ 2022 event auctioned off 38 unique items and raised $51,000 to support worldwide PyLadies chapters and future PyCon US travel grants. Join us on April 22, 2023 at the Salt Palace Convention Center and help us outdo ourselves!

The Details

• Date & Time: Saturday, April 22nd at 6:30pm
• Location: Salt Palace Convention Center at 100 West Temple St, Salt Lake City, UT 84101 in Room 151 DEFG
• Cost: 
• Entry cover charge of $35 includes dinner and a free drink ticket
• Supporter cover charge of $50.00 includes dinner, a free drink ticket, and a $15 contribution to PyLadies.
• Structure: Live bidding only (Utah state does not allow raffles, unfortunately) - but of course, no bidding required to attend!
• Register: When you register for the conference (or update your existing registration via your Dashboard), you will have an option to add a ticket to the PyLadies auction.

Donations

It wouldn't be much of an auction without items to win! If you're interested in donating an item, you can email pycon-auction@python.org or fill out the item donation form.

2022 - PSF Accounting team in action, making the auction possible!
❤️

Support for the PyLadies auction comes from attendee tickets, the PSF, and our sponsors who help cover the cost of the venue, food, and drinks. Thank you!

PyCon 2022 Chair Emily showing off her winnings from the 2022 auction with a piece made by K Lars Lohn