Category: Programming

Problem: Given a list of strings. How to convert the list to a string by concatenating all strings in the list?

Example: You want to convert list ['learn ', 'python ', 'fast'] to the string 'learn python fast'.

Solution: to convert a list of strings to a string, do the following.

• Call the ''.join(list) method on the empty string '' that glues together all strings in the list and returns a new string.
• The string on which you call the join method is used as a delimiter between the list elements.
• If you don’t need a delimiter, just use the empty string ''.

Code: Let’s have a look at the code.

lst = ['learn ', 'python ', 'fast']
print(''.join(lst))

The output is:

learn python fast

Try it yourself in our interactive Python shell:

You can also use another delimiter string, for example, the comma:

lst = ['learn' , 'python', 'fast']
print(','.join(lst))
# learn,python,fast

What Are Alternative Methods to Convert a List of Strings to a String?

Python is flexible—you can use multiple methods to achieve the same thing. So what are the different methods to convert a list to a string?

• Method 1: Use the method ''.join(list) to concatenate all strings in a given list to a single list. The string on which you call the method is the delimiter between the list elements.
• Method 2: Start with an empty string variable. Use a simple for loop to iterate over all elements in the list and add the current element to the string variable.
• Method 3: Use list comprehension [str(x) for x in list] if the list contains elements of different types to convert all elements to the string data type. Combine them using the ''.join(newlist) method.
• Method 4: Use the map function map(str, list] if the list contains elements of different types to convert all elements to the string data type. Combine them using the ''.join(newlist) method.

Here are all four variants in some code:

lst = ['learn' , 'python', 'fast'] # Method 1
print(''.join(lst))
# learnpythonfast # Method 2
s = ''
for st in lst: s += st
print(s)
# learnpythonfast # Method 3
lst = ['learn', 9, 'python', 9, 'fast']
s = ''.join([str(x) for x in lst])
print(s)
# learn9python9fast # Method 4
lst = ['learn', 9, 'python', 9, 'fast']
s = ''.join(map(str, lst))
print(s)
# learn9python9fast

Again, try to modify the delimiter string yourself using our interactive code shell:

So far so good. You’ve learned how to convert a list to a string. But that’s not all! Let’s dive into some more specifics of converting a list to a string.

Python List to String with Commas

Problem: Given a list of strings. How to convert the list to a string by concatenating all strings in the list—using a comma as the delimiter between the list elements?

Example: You want to convert list ['learn', 'python', 'fast'] to the string 'learn,python,fast'.

Solution: to convert a list of strings to a string, call the ','.join(list) method on the delimiter string ',' that glues together all strings in the list and returns a new string.

Code: Let’s have a look at the code.

lst = ['learn', 'python', 'fast']
print(','.join(lst))

The output is:

learn,python,fast

Python List to String with Spaces

Problem: Given a list of strings. How to convert the list to a string by concatenating all strings in the list—using a space as the delimiter between the list elements?

Example: You want to convert list ['learn', 'python', 'fast'] to the string 'learn python fast'. (Note the empty spaces between the terms.)

Solution: to convert a list of strings to a string, call the ' '.join(list) method on the string ' ' (space character) that glues together all strings in the list and returns a new string.

Code: Let’s have a look at the code.

lst = ['learn', 'python', 'fast']
print(' '.join(lst))

The output is:

learn python fast

Python List to String with Newlines

Problem: Given a list of strings. How to convert the list to a string by concatenating all strings in the list—using a newline character as the delimiter between the list elements?

Example: You want to convert list ['learn', 'python', 'fast'] to the string 'learn\npython\nfast' or as a multiline string:

'''learn
python
fast'''

Solution: to convert a list of strings to a string, call the '\n'.join(list) method on the newline character '\n' that glues together all strings in the list and returns a new string.

Code: Let’s have a look at the code.

lst = ['learn', 'python', 'fast']
print('\n'.join(lst))

The output is:

learn
python
fast

Python List to String with Quotes

Problem: Given a list of strings. How to convert the list to a string by concatenating all strings in the list—using a comma character followed by an empty space as the delimiter between the list elements? Additionally, you want to wrap each string in double quotes.

Example: You want to convert list ['learn', 'python', 'fast'] to the string '"learn", "python", "fast"' :

Solution: to convert a list of strings to a string, call the ', '.join('"' + x + '"' for x in lst) method on the delimiter string ', ' that glues together all strings in the list and returns a new string. You use a generator expression to modify each element of the original element so that it is enclosed by the double quote " chararacter.

Code: Let’s have a look at the code.

lst = ['learn', 'python', 'fast']
print(', '.join('"' + x + '"' for x in lst))

The output is:

"learn", "python", "fast"

Python List to String with Brackets

Problem: Given a list of strings. How to convert the list to a string by concatenating all strings in the list—using a comma character followed by an empty space as the delimiter between the list elements? Additionally, you want to wrap the whole string in a square bracket to indicate that’s a list.

Example: You want to convert list ['learn', 'python', 'fast'] to the string '[learn, python, fast]' :

Solution: to convert a list of strings to a string, call the '[' + ', '.join(lst) + ']' method on the delimiter string ', ' that glues together all strings in the list and returns a new string.

Code: Let’s have a look at the code.

lst = ['learn', 'python', 'fast']
print('[' + ', '.join(lst) + ']')

The output is:

[learn, python, fast]

Python List to String and Back

Problem: You want to convert a list into a string and back to a list.

Example: Convert the list ['learn', 'python', 'fast'] to a string "['learn', 'python', 'fast']" and back to a list ['learn', 'python', 'fast'].

Solution: Use the following two steps to convert back and forth between a string and a list:

• Use the built-in str() function to convert from a list to a string.
• Use the built-in eval() function to convert from a string to a list.

lst = ['learn', 'python', 'fast'] converted = str(lst)
print(converted)
# ['learn', 'python', 'fast']
print(type(converted))
# <class 'str'> original = eval(converted)
print(original)
# ['learn', 'python', 'fast']
print(type(original))
# <class 'list'>

Although the output of both the converted list and the original list look the same, you can see that the data type is string for the former and list for the latter.

Convert List of Int to String

Problem: You want to convert a list into a string but the list contains integer values.

Example: Convert the list [1, 2, 3] to a string '123'.

Solution: Use the join method in combination with a generator expression to convert the list of integers to a single string value:

lst = [1, 2, 3]
print(''.join(str(x) for x in lst))
# 123

The generator expression converts each element in the list to a string. You can then combine the string elements using the join method of the string object.

If you miss the conversion from integer to string, you get the following TypeError:

lst = [1, 2, 3]
print(''.join(lst)) '''
Traceback (most recent call last): File "C:\Users\xcent\Desktop\code.py", line 2, in <module> print(''.join(lst))
TypeError: sequence item 0: expected str instance, int found '''

Python List to String One Line

To convert a list to a string in one line, use either of the three methods:

• Use the ''.join(list) method to glue together all list elements to a single string.
• Use the list comprehension method [str(x) for x in lst] to convert all list elements to type string.
• Use str(list) to convert the list to a string representation.

Here are three examples:

lst = ['finxter', 'is', 'awesome']
print(' '.join(lst))
# finxter is awesome lst = [1, 2, 3]
print([str(x) for x in lst])
# ['1', '2', '3'] print(str(lst))
# [1, 2, 3]

Where to Go From Here

Want to increase your Python skill on a daily basis? Just by following a series of FREE Python course emails? Then join the #1 Python Email Academy in the world!

For my subscribers, I regularly publish educative emails about the most important Python topics. Register and join my community of thousands of ambitious coders. I guarantee, you will love it!

(Besides—it’s free and you can unsubscribe anytime so you’ve nothing to lose and everything to gain.)

join the #1 Python Email Academy in the world!

[Spoiler] Which function filters a list faster: filter() vs list comprehension? For large lists with one million elements, filtering lists with list comprehension is 40% faster than the built-in filter() method.

To answer this question, I’ve written a short script that tests the runtime performance of filtering large lists of increasing sizes using the filter() and the list comprehension methods.

My thesis is that the list comprehension method should be slightly faster for larger list sizes because it leverages the efficient cPython implementation of list comprehension and doesn’t need to call an extra function.

Related Article:

• How to Filter a List in Python?

I used my notebook with an Intel(R) Core(TM) i7-8565U 1.8GHz processor (with Turbo Boost up to 4.6 GHz) and 8 GB of RAM.

Try It Yourself:

import time # Compare runtime of both methods
list_sizes = [i * 10000 for i in range(100)]
filter_runtimes = []
list_comp_runtimes = [] for size in list_sizes: lst = list(range(size)) # Get time stamps time_0 = time.time() list(filter(lambda x: x%2, lst)) time_1 = time.time() [x for x in lst if x%2] time_2 = time.time() # Calculate runtimes filter_runtimes.append((size, time_1 - time_0)) list_comp_runtimes.append((size, time_2 - time_1)) # Plot everything
import matplotlib.pyplot as plt
import numpy as np f_r = np.array(filter_runtimes)
l_r = np.array(list_comp_runtimes) print(filter_runtimes)
print(list_comp_runtimes) plt.plot(f_r[:,0], f_r[:,1], label='filter()')
plt.plot(l_r[:,0], l_r[:,1], label='list comprehension') plt.xlabel('list size')
plt.ylabel('runtime (seconds)') plt.legend()
plt.savefig('filter_list_comp.jpg')
plt.show()

The code compares the runtimes of the filter() function and the list comprehension variant to filter a list. Note that the filter() function returns a filter object, so you need to convert it to a list using the list() constructor.

Here’s the resulting plot that compares the runtime of the two methods. On the x axis, you can see the list size from 0 to 1,000,000 elements. On the y axis, you can see the runtime in seconds needed to execute the respective functions.

The resulting plot shows that both methods are extremely fast for a few tens of thousands of elements. In fact, they are so fast that the time() function of the time module cannot capture the elapsed time.

But as you increase the size of the lists to hundreds of thousands of elements, the list comprehension method starts to win:

For large lists with one million elements, filtering lists with list comprehension is 40% faster than the built-in filter() method.

The reason is the efficient implementation of the list comprehension statement. An interesting observation is the following though. If you don’t convert the filter function to a list, you get the following result:

Suddenly the filter() function has constant runtime of close to 0 seconds—no matter how many elements are in the list. Why is this happening?

The explanation is simple: the filter function returns an iterator, not a list. The iterator doesn’t need to compute a single element until it is requested to compute the next() element. So, the filter() function computes the next element only if it is required to do so. Only if you convert it to a list, it must compute all values. Otherwise, it doesn’t actually compute a single value beforehand.

Where to Go From Here

This tutorial has shown you the filter() function in Python and compared it against the list comprehension way of filtering: [x for x in list if condition]. You’ve seen that the latter is not only more readable and more Pythonic, but also faster. So take the list comprehension approach to filter lists!

If you love coding and you want to do this full-time from the comfort of your own home, you’re in luck:

I’ve created a free webinar that shows you how I started as a Python freelancer after my computer science studies working from home (and seeing my kids grow up) while earning a full-time income working only part-time hours.

Webinar: How to Become Six-Figure Python Freelancer?

Join 21,419 ambitious Python coders. It’s fun!

Nested List Comprehension—what does it even mean?

There are three equally interpretations of this term:

• Coming from a computer science background, I was assuming that “nested list comprehension” refers to the creation of a list of lists. In other words: How to create a nested list with list comprehension?
• But after a bit of research, I learned that there is a second interpretation of nested list comprehension: How to use a nested for loop in the list comprehension?
• A few months later, I realized that some people use “nested list comprehension” to mean the use of a list comprehension statement as expression within a list comprehension statement. In other words: How to use a list comprehension statement within a list comprehension statement? (Watch the video to learn about this third interpretation.)

How to Create a Nested List with List Comprehension?

It is possible to create a nested list with list comprehension in Python. What is a nested list? It’s a list of lists. Here is an example:

## Nested List Comprehension
lst = [[x for x in range(5)] for y in range(3)]
print(lst)
# [[0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4]]

As you can see, we create a list with three elements. Each list element is a list by itself.

Everything becomes clear when we go back to our magic formula of list comprehension: [ expression + context]. The expression part generates a new list consisting of 5 integers. The context part repeats this three times. Hence, each of the three nested lists has five elements.

If you are an advanced programmer, you may ask whether there is some aliasing going on here. Aliasing in this context means that the three list elements point to the same list [0, 1, 2, 3, 4]. This is not the case because each expression is evaluated separately, a new list is created for each of the three context executions. This is nicely demonstrated in this code snippet:

l[0].append(5)
print(l)
# [[0, 1, 2, 3, 4, 5], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4]]
# ... and not [[0, 1, 2, 3, 4, 5], [0, 1, 2, 3, 4, 5], [0, 1, 2, 3, 4, 5]]

How to Use a Nested For Loop in the List Comprehension?

To be frank, this is super-simple stuff. Do you remember the formula of list comprehension (= ‘[‘ + expression + context + ‘]’)?

The context is an arbitrary complex restriction construct of for loops and if restrictions with the goal of specifying the data items on which the expression should be applied.

In the expression, you can use any variable you define within a for loop in the context. Let’s have a look at an example.

Suppose you want to use list comprehension to make this code more concise (for example, you want to find all possible pairs of users in your social network application):

# BEFORE
users = ["John", "Alice", "Ann", "Zach"]
pairs = []
for x in users: for y in users: if x != y: pairs.append((x,y))
print(pairs)
#[('John', 'Alice'), ('John', 'Ann'), ('John', 'Zach'), ('Alice', 'John'), ('Alice', 'Ann'), ('Alice', 'Zach'), ('Ann', 'John'), ('Ann', 'Alice'), ('Ann', 'Zach'), ('Zach', 'John'), ('Zach', 'Alice'), ('Zach', 'Ann')]

Now, this code is a mess! How can we fix it? Simply use nested list comprehension!

# AFTER
pairs = [(x,y) for x in users for y in users if x!=y]
print(pairs)
# [('John', 'Alice'), ('John', 'Ann'), ('John', 'Zach'), ('Alice', 'John'), ('Alice', 'Ann'), ('Alice', 'Zach'), ('Ann', 'John'), ('Ann', 'Alice'), ('Ann', 'Zach'), ('Zach', 'John'), ('Zach', 'Alice'), ('Zach', 'Ann')]

As you can see, we are doing exactly the same thing as with un-nested list comprehension. The only difference is to write the two for loops and the if statement in a single line within the list notation [].

How to Use a List Comprehension Statement Within a List Comprehension Statement?

Our goal is to solve the following problem: given a multiline string, create a list of lists—each consisting of all the words in a line that have more than three characters.

## Data
text = '''
Call me Ishmael. Some years ago - never mind how long precisely - having
little or no money in my purse, and nothing particular to interest me
on shore, I thought I would sail about a little and see the watery part
of the world. It is a way I have of driving off the spleen, and regulating
the circulation. - Moby Dick''' words = [[x for x in line.split() if len(x)>3] for line in text.split('\n')] print(words)

This is a nested list comprehension statement. This creates a new inner list as an element of the outer list. Each inner list contains all words with more than 4 characters. Each outer list contains an inner list for each line of text.

Where to Go From Here?

Want to increase your Python skill on a daily basis? Just by following a series of FREE Python course emails? Then join the #1 Python Email Academy in the world!

For my subscribers, I regularly publish educative emails about the most important Python topics. Register and join my community of thousands of ambitious coders. I guarantee, you will love it!

(Besides—it’s free and you can unsubscribe anytime so you’ve nothing to lose and everything to gain.)

join the #1 Python Email Academy in the world!

In this article, you’ll learn the ins and outs of the sorting function in Python. In particular, you’re going to learn how to filter a list of dictionaries. So let’s get started!

Short answer: The list comprehension statement [x for x in lst if condition(x)] creates a new list of dictionaries that meet the condition. All dictionaries in lst that don’t meet the condition are filtered out. You can define your own condition on list element x.

Here’s a quick and minimal example:

l = [{'key':10}, {'key':4}, {'key':8}] def condition(dic): ''' Define your own condition here''' return dic['key'] > 7 filtered = [d for d in l if condition(d)] print(filtered)
# [{'key': 10}, {'key': 8}]

Try it yourself in the interactive Python shell (in your browser):

You’ll now get the step-by-step solution of this solution. I tried to keep it as simple as possible. So keep reading!

Filter a List of Dictionaries By Value

Problem: Given a list of dictionaries. Each dictionary consists of one or more (key, value) pairs. You want to filter them by value of a particular dictionary key (attribute). How do you do this?

Minimal Example: Consider the following example where you’ve three user dictionaries with username, age, and play_time keys. You want to get a list of all users that meet a certain condition such as play_time>100. Here’s what you try to accomplish:

users = [{'username': 'alice', 'age': 23, 'play_time': 101}, {'username': 'bob', 'age': 31, 'play_time': 88}, {'username': 'ann', 'age': 25, 'play_time': 121},] superplayers = # Filtering Magic Here print(superplayers)

The output should look like this where the play_time attribute determines whether a dictionary passes the filter or not, i.e., play_time>100:

[{'username': 'alice', 'age': 23, 'play_time': 101}, {'username': 'ann', 'age': 25, 'play_time': 121}]

Solution: Use list comprehension [x for x in lst if condition(x)] to create a new list of dictionaries that meet the condition. All dictionaries in lst that don’t meet the condition are filtered out. You can define your own condition on list element x.

Here’s the code that shows you how to filter out all user dictionaries that don’t meet the condition of having played at least 100 hours.

users = [{'username': 'alice', 'age': 23, 'play_time': 101}, {'username': 'bob', 'age': 31, 'play_time': 88}, {'username': 'ann', 'age': 25, 'play_time': 121},] superplayers = [user for user in users if user['play_time']>100] print(superplayers)

The output is the filtered list of dictionaries that meet the condition:

[{'username': 'alice', 'age': 23, 'play_time': 101}, {'username': 'ann', 'age': 25, 'play_time': 121}]

Try It Yourself:

Related articles on the Finxter blog:

• List Comprehension
• Lambda Functions
• Dictionaries

Filter a List of Dictionaries By Key

Problem: Given a list of dictionaries. Each dictionary consists of one or more (key, value) pairs. You want to filter them by key (attribute). All dictionaries that don’t have this key (attribute) should be filtered out. How do you do this?

Minimal Example: Consider the following example again where you’ve three user dictionaries with username, age, and play_time keys. You want to get a list of all users for which the key play_time exists. Here’s what you try to accomplish:

users = [{'username': 'alice', 'age': 23, 'play_time': 101}, {'username': 'bob', 'age': 31, 'play_time': 88}, {'username': 'ann', 'age': 25},] superplayers = # Filtering Magic Here print(superplayers)

The output should look like this where the play_time attribute determines whether a dictionary passes the filter or not (as long as it exists, it shall pass the filter).

[{'username': 'alice', 'age': 23, 'play_time': 101}, {'username': 'bob', 'age': 31, 'play_time': 88}]

Solution: Use list comprehension [x for x in lst if condition(x)] to create a new list of dictionaries that meet the condition. All dictionaries in lst that don’t meet the condition are filtered out. You can define your own condition on list element x.

Here’s the code that shows you how to filter out all user dictionaries that don’t meet the condition of having a key play_time.

users = [{'username': 'alice', 'age': 23, 'play_time': 101}, {'username': 'bob', 'age': 31, 'play_time': 88}, {'username': 'ann', 'age': 25},] superplayers = [user for user in users if 'play_time' in user] print(superplayers)

The output is the filtered list of dictionaries that meet the condition:

[{'username': 'alice', 'age': 23, 'play_time': 101}, {'username': 'bob', 'age': 31, 'play_time': 88}]

Try It Yourself:

Related articles on the Finxter blog:

• List Comprehension
• Lambda Functions
• Dictionaries

Related Question: How to Sort a List of Dictionaries By Key Value

Original article

Problem: Given a list of dictionaries. Each dictionary consists of multiple (key, value) pairs. You want to sort them by value of a particular dictionary key (attribute). How do you sort this dictionary?

Minimal Example: Consider the following example where you want to sort a list of salary dictionaries by value of the key 'Alice'.

salaries = [{'Alice': 100000, 'Bob': 24000}, {'Alice': 121000, 'Bob': 48000}, {'Alice': 12000, 'Bob': 66000}] sorted_salaries = # ... Sorting Magic Here ... print(sorted_salaries)

The output should look like this where the salary of Alice determines the order of the dictionaries:

[{'Alice': 12000, 'Bob': 66000},
{'Alice': 100000, 'Bob': 24000},
{'Alice': 121000, 'Bob': 48000}]

Solution: You have two main ways to do this—both are based on defining the key function of Python’s sorting methods. The key function maps each list element (in our case a dictionary) to a single value that can be used as the basis of comparison.

• Use a lambda function as key function to sort the list of dictionaries.
• Use the itemgetter function as key function to sort the list of dictionaries.

Here’s the code of the first option using a lambda function that returns the value of the key 'Alice' from each dictionary:

# Create the dictionary of Bob's and Alice's salary data
salaries = [{'Alice': 100000, 'Bob': 24000}, {'Alice': 121000, 'Bob': 48000}, {'Alice': 12000, 'Bob': 66000}] # Use the sorted() function with key argument to create a new dic.
# Each dictionary list element is "reduced" to the value stored for key 'Alice'
sorted_salaries = sorted(salaries, key=lambda d: d['Alice']) # Print everything to the shell
print(sorted_salaries)

The output is the sorted dictionary. Note that the first dictionary has the smallest salary of Alice and the third dictionary has the largest salary of Alice.

[{'Alice': 12000, 'Bob': 66000}, {'Alice': 100000, 'Bob': 24000}, {'Alice': 121000, 'Bob': 48000}]

Try It Yourself:

Related articles:

• The Ultimate Guide to Python Lists

Where to Go From Here

In this article, you’ve learned how to filter a list of dictionaries easily with a simple list comprehension statement. That’s far more efficient than using the filter() method proposed in many other blog tutorials. Guido, the creator of Python, hated the filter() function!

I’ve realized that professional coders tend to use dictionaries more often than beginners due to their superior understanding of the benefits of dictionaries. If you want to learn about those, check out my in-depth tutorial of Python dictionaries.

If you want to stop learning and start earning with Python, check out my free webinar “How to Become a Python Freelance Developer?”. It’s a great way of starting your thriving coding business online.

“[Webinar] How to Become a Python Freelance Developer?”

Here’s your free PDF cheat sheet showing you all Python list methods on one simple page. Click the image to download the high-resolution PDF file, print it, and post it to your office wall:

I’ll lead you through all Python list methods in this short video tutorial:

If you want to study the methods yourself, have a look at the following table from my blog article:

Method	Description
lst.append(x)	Appends element x to the list lst.
lst.clear()	Removes all elements from the list lst–which becomes empty.
lst.copy()	Returns a copy of the list lst. Copies only the list, not the elements in the list (shallow copy).
lst.count(x)	Counts the number of occurrences of element x in the list lst.
lst.extend(iter)	Adds all elements of an iterable iter(e.g. another list) to the list lst.
lst.index(x)	Returns the position (index) of the first occurrence of value x in the list lst.
lst.insert(i, x)	Inserts element x at position (index) i in the list lst.
lst.pop()	Removes and returns the final element of the list lst.
lst.remove(x)	Removes and returns the first occurrence of element x in the list lst.
lst.reverse()	Reverses the order of elements in the list lst.
lst.sort()	Sorts the elements in the list lst in ascending order.

Go ahead and try the Python list methods yourself:

Puzzle: Can you figure out all outputs of this interactive Python script?

If you’ve studied the table carefully, you’ll know the most important list methods in Python. Let’s have a look at some examples of above methods:

>>> l = []
>>> l.append(2)
>>> l
[2]
>>> l.clear()
>>> l
[]
>>> l.append(2)
>>> l
[2]
>>> l.copy()
[2]
>>> l.count(2)
1
>>> l.extend([2,3,4])
>>> l
[2, 2, 3, 4]
>>> l.index(3)
2
>>> l.insert(2, 99)
>>> l
[2, 2, 99, 3, 4]
>>> l.pop()
4
>>> l.remove(2)
>>> l
[2, 99, 3]
>>> l.reverse()
>>> l
[3, 99, 2]
>>> l.sort()
>>> l
[2, 3, 99]

Where to Go From Here?

Want more cheat sheets? Excellent. I believe learning with cheat sheets is one of the most efficient learning techniques. Join my free Python email list where I’ll send you more than 10 new Python cheat sheets and regular Python courses for continuous improvement. It’s free!

In this article, I’ll show you how to divide a list into equally-sized chunks in Python. Step-by-step, you’ll arrive at the following great code that accomplishes exactly that:

You can play around with the code yourself but if you need some explanations, read on because I’ll explain it to you in much detail:

Chunking Your List

Let’s make this question more palpable by transforming it into a practical problem:

Problem: Imagine that you have a temperature sensor that sends data every 6 minutes, which makes 10 data points per hour. All these data points are stored in one list for each day.

Now, we want to have a list of hourly average temperatures for each day—this is why we need to split the list of data for one day into evenly sized chunks.

Solution: To achieve this, we use a for-loop and Python’s built-in function range() which we have to examine in depth.

The range() function can be used either with one, two or three arguments.

• If you use it with one single argument, e.g., range(10), we get a range object containing the numbers 0 to 9. So, if you call range with one argument, this argument will be interpreted as the max or stop value of the range, but it is excluded from the range.
• You can also call the range() function with two arguments, e.g., range(5, 10). This call with two arguments returns a range object containing the numbers 5 to 9. So, now we have a lower and an upper bound for the range. Contrary to the stop value, the start value is included in the range.
• In a call of the function range() with three parameters, the first parameter is the start value, the second one is the stop value and the third value is the step size. For example, range(5, 15, 2) returns a range object containing the following values: 5, 7, 9, 11, 13. As you can see, the range starts with the start and then it adds the step value as long as the values are less than the stop value.

In our problem, our chunks have a length of 10, the start value is 0 and the max value is the end of the list of data.

Putting all together: Calling range(0, len(data), 10) will give us exactly what we need to iterate over the chunks. Let’s put some numbers there to visualize it.

For one single day, we have a data length of 24 * 10 = 240, so the call of the range function would be this: range(0, 240, 10) and the resulting range would be 0, 10, 20, 30, …, 230. Pause a moment and consider these values: they represent the indices of the first element of each chunk.

So what do we have now? The start indices of each chunk and also the length – and that’s all we need to slice the input data into the chunks we need.

The slicing operator takes two or three arguments separated by the colon : symbol. They have the same meaning as in the range function.

If you want to know more about slicing read our detailed article here.

A first draft of our code could be this:

data = [15.7, 16.2, 16.5, 15.9, ..., 27.3, 26.4, 26.1, 27.2]
chunk_length = 10 for i in range(0, len(data), chunk_length): print(data[i:i+chunk_length])

Play with this code in our interactive Python shell:

However, we can still improve this code and make it reusable by creating a generator out of it.

Chunking With Generator Expressions

A generator is a function but instead of a return statement it uses the keyword yield.

The keyword yield interrupts the function and returns a value. The next time the function gets called, the next value is returned and the function’s execution stops again. This behavior can be used in a for-loop, where we want to get a value from the generator, work with this value inside the loop and then repeat it with the next value. Now, let’s take a look at the improved version of our code:

data = [15.7, 16.2, 16.5, 15.9, ..., 27.3, 26.4, 26.1, 27.2]
chunk_length = 10 def make_chunks(data, length): for i in range(0, len(data), length): yield data[i:i + length] for chunk in make_chunks(data, chunk_length): print(chunk)

That looks already pretty pythonic and we can reuse the function make_chunks() for all the other data we need to process.

Let’s finish the code so that we get a list of hourly average temperatures as result.

import random def make_chunks(data, length): for i in range(0, len(data), length): yield data[i:i + length] def process(chunk): return round(sum(chunk)/len(chunk), 2) n = 10
# generate random temperature values
day_temperatures = [random.random() * 20 for x in range(24 * n)]
avg_per_hour = [] for chunk in make_chunks(day_temperatures, n): r = process(batch) avg_per_hour.append(r) print(avg_per_hour)

And that’s it, this cool pythonic code solves our problem. We can make the code even a bit shorter but I consider this code less readable because you need to know really advanced Python concepts.

import random make_chunks = lambda data, n: (data[i:i + n] for i in range(0, len(data), n))
process = lambda data: round(sum(data)/len(data), 2) n = 10
# generate random temperature values
day_temperatures = [random.random() * 20 for x in range(24 * n)]
avg_per_hour = [] for chunk in make_chunks(day_temperatures, n): r = process(batch) avg_per_hour.append(r) print(avg_per_hour)

So, what did we do? We reduced the helper functions to lambda expressions and for the generator function we use a special shorthand – the parenthesis.

Summary

To sum up the solution: We used the range function with three arguments, the start value, the stop value and the step value. By setting the step value to our desired chunk length, the start value to 0 and the stop value to the total data length, we get a range object containing all the start indices of our chunks. With the help of slicing we can access exactly the chunk we need in each iteration step.

Where to Go From Here?

Want to start earning a full-time income with Python—while working only part-time hours? Then join our free Python Freelancer Webinar.

It shows you exactly how you can grow your business and Python skills to a point where you can work comfortable for 3-4 hours from home and enjoy the rest of the day (=20 hours) spending time with the persons you love doing things you enjoy to do.

Become a Python freelancer now!

Want to calculate the standard deviation of a column in your Pandas DataFrame?

In case you’ve attended your last statistics course a few years ago, let’s quickly recap the definition of variance: it’s the average squared deviation of the list elements from the average value.

You can do this by using the pd.std() function that calculates the standard deviation along all columns. You can then get the column you’re interested in after the computation.

import pandas as pd # Create your Pandas DataFrame
d = {'username': ['Alice', 'Bob', 'Carl'], 'age': [18, 22, 43], 'income': [100000, 98000, 111000]}
df = pd.DataFrame(d) print(df)

Your DataFrame looks like this:

	username	age	income
0	Alice	18	100000
1	Bob	22	98000
2	Carl	43	111000

Here’s how you can calculate the standard deviation of all columns:

print(df.std())

The output is the standard deviation of all columns:

age 13.428825
income 7000.000000
dtype: float64

To get the variance of an individual column, access it using simple indexing:

print(df.std()['age'])
# 180.33333333333334

Together, the code looks as follows. Use the interactive shell to play with it!

Standard Deviation in NumPy Library

Python’s package for data science computation NumPy also has great statistics functionality. You can calculate all basic statistics functions such as average, median, variance, and standard deviation on NumPy arrays. Simply import the NumPy library and use the np.var(a) method to calculate the average value of NumPy array a.

Here’s the code:

import numpy as np a = np.array([1, 2, 3])
print(np.std(a))
# 0.816496580927726

Where to Go From Here?

Before you can become a data science master, you first need to master Python. Join my free Python email course and receive your daily Python lesson directly in your INBOX. It’s fun!

Join The World’s #1 Python Email Academy [+FREE Cheat Sheets as PDF]

Imagine the following scenario:

You work in law enforcement for the US Department of Labor, finding companies that pay below minimum wage so you can initiate further investigations. Like hungry dogs on the back of a meat truck, your Fair Labor Standards Act (FLSA) officers are already waiting for the list of companies that violated the minimum wage law. Can you give it to them?

Here’s the code from my new book “Python One-Liners“:

companies = {'CoolCompany' : {'Alice' : 33, 'Bob' : 28, 'Frank' : 29}, 'CheapCompany' : {'Ann' : 4, 'Lee' : 9, 'Chrisi' : 7}, 'SosoCompany' : {'Esther' : 38, 'Cole' : 8, 'Paris' : 18}} illegal = [x for x in companies if any(y<9 for y in companies[x].values())] print(illegal)

Try it yourself in our interactive Python shell: