11-04-2020, 11:44 PM
Pandas NaN — Working With Missing Data
<div><p><em>Pandas is Excel on steroids—the powerful Python library allows you to analyze structured and tabular data with surprising efficiency and ease. Pandas is one of the reasons why master coders reach 100x the efficiency of average coders. In today’s article, you’ll learn how to work with missing data—in particular, how to <strong>handle NaN values</strong> in <a href="https://blog.finxter.com/pandas-cheat-sheets/" title="[PDF Collection] 7 Beautiful Pandas Cheat Sheets — Post Them to Your Wall" target="_blank" rel="noreferrer noopener">Pandas DataFrames</a>.</em></p>
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio">
<div class="wp-block-embed__wrapper">
<div class="ast-oembed-container"><iframe title="PANDAS ULTIMATE GUIDE TO NAN (NOT A NUMBER)" width="1400" height="788" src="https://www.youtube.com/embed/ihslLmCnlTw?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>
</div>
</figure>
<p>You’ll learn about all the different reasons why NaNs appear in your DataFrames—and how to handle them. Let’s get started!</p>
<h2>Checking Series for NaN Values</h2>
<p><em><strong>Problem</strong>: How to check a series for NaN values?</em></p>
<p>Have a look at the following code:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd
import numpy as np data = pd.Series([0, np.NaN, 2])
result = data.hasnans print(result)
# True</pre>
<p>Series can contain <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-values—an abbreviation for <em>Not-A-Number</em>—that describe undefined values.</p>
<p>To check if a Series contains one or more <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value, use the attribute <code data-enlighter-language="generic" class="EnlighterJSRAW">hasnans</code>. The attribute returns <code data-enlighter-language="generic" class="EnlighterJSRAW">True</code> if there is at least one <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value and <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code> otherwise. </p>
<p>There’s a <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value in the Series, so the output is <code data-enlighter-language="generic" class="EnlighterJSRAW">True</code>.</p>
<h2>Filtering Series Generates NaN</h2>
<p><em><strong>Problem</strong>: When <a href="https://blog.finxter.com/how-to-filter-a-list-in-python/" title="How to Filter a List in Python?" target="_blank" rel="noreferrer noopener">filtering </a>a Series with <code data-enlighter-language="generic" class="EnlighterJSRAW">where()</code> and no element passes the filtering condition, what’s the result?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd xs = pd.Series([5, 1, 4, 2, 3])
xs.where(xs > 2, inplace=True)
result = xs.hasnans print(result)
# True</pre>
<p>The method <code data-enlighter-language="generic" class="EnlighterJSRAW">where()</code> filters a Series by a <a href="https://blog.finxter.com/how-to-conditionally-select-elements-in-a-numpy-array/" title="How to Conditionally Select Elements in a Numpy Array?" target="_blank" rel="noreferrer noopener">condition</a>. Only the elements that satisfy the condition remain in the resulting Series. And what happens if a value doesn’t satisfy the condition? Per default, all rows not satisfying the condition are filled with <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-values.</p>
<p>This is why our Series contains <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-values after filtering it with the method <code data-enlighter-language="generic" class="EnlighterJSRAW">where()</code>.</p>
<h2>Working with Multiple Series of Different Lengths</h2>
<p><em><strong>Problem</strong>: If you element-wise add two Series objects with a different number of elements—what happens with the remaining elements?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd s = pd.Series(range(0, 10))
t = pd.Series(range(0, 20))
result = (s + t)[1] print(result)
# 2</pre>
<p>To add two Series element-wise, use the default addition operator <code data-enlighter-language="generic" class="EnlighterJSRAW">+</code>. The Series do not need to have the same size because once the first Series ends, the subsequent element-wise results are <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values.</p>
<p>At index <code data-enlighter-language="generic" class="EnlighterJSRAW">1</code> in the resulting Series, you get the result of <code data-enlighter-language="generic" class="EnlighterJSRAW">1 + 1 = 2</code>.</p>
<h2>Create a DataFrame From a List of Dictionaries with Unequal Keys</h2>
<p><em><strong>Problem</strong>: How to create a DataFrame from a list of dictionaries if the dictionaries have unequal keys? A DataFrame expects the same columns to be available for each row!</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd data = [{'Car':'Mercedes', 'Driver':'Hamilton, Lewis'}, {'Car':'Ferrari', 'Driver':'Schumacher, Michael'}, {'Car':'Lamborghini'}] df = pd.DataFrame(data, index=['Rank 2', 'Rank 1', 'Rank 3'])
df.sort_index(inplace=True)
result = df['Car'].iloc[0] print(result)
# Ferrari</pre>
<p>You can create a DataFrame from a <a href="https://blog.finxter.com/how-to-create-a-list-of-dictionaries-in-python/" target="_blank" rel="noreferrer noopener" title="How to Create a List of Dictionaries in Python?">list of dictionaries</a>. The dictionaries’ keys define the column labels, and the values define the columns’ entries. Not all dictionaries must contain the same keys. If a dictionary doesn’t contain a particular key, this will be interpreted as a <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-value.</p>
<p>This code snippet uses string labels as index values to <a href="https://blog.finxter.com/how-to-sort-a-dictionary-by-value-in-python/" title="How To Sort A Dictionary By Value in Python?" target="_blank" rel="noreferrer noopener">sort </a>the DataFrame. After sorting the DataFrame, the row with index label <code data-enlighter-language="generic" class="EnlighterJSRAW">Rank 1</code> is at location <code data-enlighter-language="generic" class="EnlighterJSRAW">0</code> in the DataFrame and the value in the column <code data-enlighter-language="generic" class="EnlighterJSRAW">Car</code> is <code data-enlighter-language="generic" class="EnlighterJSRAW">Ferrari</code>.</p>
<h2>Sorting a DataFrame by Column with NaN Values</h2>
<p><em><strong>Problem</strong>: What happens if you sort a DataFrame by column if the column contains a <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- selection = df.sort_values(by="engine-size")
result = selection.index.to_list()[0]
print(result)
# 1</pre>
<p>In this code snippet, you sort the rows of the DataFrame by the values of the column <code data-enlighter-language="generic" class="EnlighterJSRAW">engine-size</code>.</p>
<p>The main point is that <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values are always moved to the end in Pandas sorting. Thus, the first value is <code data-enlighter-language="generic" class="EnlighterJSRAW">1.8</code>, which belongs to the row with index value <code data-enlighter-language="generic" class="EnlighterJSRAW">1</code>.</p>
<h2>Count Non-NaN Values</h2>
<p><em><strong>Problem</strong>: How to count the number of elements in a dataframe column that are not <code data-enlighter-language="generic" class="EnlighterJSRAW">Nan</code>?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- df.count()[5]
print(result)
# 4</pre>
<p>The method <code data-enlighter-language="generic" class="EnlighterJSRAW">count()</code> returns the number of non-<code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values for each column. The DataFrame <code data-enlighter-language="generic" class="EnlighterJSRAW">df</code> has five rows. The fifth column<br />contains one <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value. Therefore, the count of the fifth column is <code data-enlighter-language="generic" class="EnlighterJSRAW">4</code>.</p>
<h2>Drop NaN-Values</h2>
<p><em><strong>Problem</strong>: How to drop all rows that contain a <code>NaN</code> value in any of its columns—and how to restrict this to certain columns?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- selection1 = df.dropna(subset=["price"])
selection2 = df.dropna()
print(len(selection1), len(selection2))
# 5 4</pre>
<p>The DataFrame’s <code data-enlighter-language="generic" class="EnlighterJSRAW">dropna()</code> method drops all rows that contain a <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value in any of its columns. But how to restrict the columns to be scanned for <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values?</p>
<p>By passing a list of column labels to the optional parameter <code data-enlighter-language="generic" class="EnlighterJSRAW">subset</code>, you can define which columns you want to consider.</p>
<p>The call of <code data-enlighter-language="generic" class="EnlighterJSRAW">dropna()</code> without restriction, drops line <code data-enlighter-language="generic" class="EnlighterJSRAW">2</code> because of the <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value in the column <code data-enlighter-language="generic" class="EnlighterJSRAW">engine-size</code>. When you restrict the columns only to <code data-enlighter-language="generic" class="EnlighterJSRAW">price</code>, no rows will be dropped, because no <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value is present.</p>
<h2>Drop Nan and Reset Index</h2>
<p><em><strong>Problem</strong>: What happens to indices after dropping certain rows?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- df.drop([0, 1, 2], inplace=True)
df.reset_index(inplace=True)
result = df.index.to_list()
print(result)
# [0, 1]</pre>
<p>The method <code data-enlighter-language="generic" class="EnlighterJSRAW">drop()</code> on a DataFrame deletes rows or columns by index. You can either pass a single value or a list of values.</p>
<p>By default the <code data-enlighter-language="generic" class="EnlighterJSRAW">inplace</code> parameter is set to <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code>, so that modifications won’t affect the initial DataFrame object. Instead, the method returns a modified copy of the DataFrame. In the puzzle, you set <code data-enlighter-language="generic" class="EnlighterJSRAW">inplace</code> to <code data-enlighter-language="generic" class="EnlighterJSRAW">True</code>, so the deletions are performed directly on the DataFrame.</p>
<p>After deleting the first three rows, the first two index labels are 3 and 4. You can reset the default indexing by calling the method <code data-enlighter-language="generic" class="EnlighterJSRAW">reset_index()</code> on the DataFrame, so that the index starts at 0 again. As there are only two rows left in the DataFrame, the result is <code data-enlighter-language="generic" class="EnlighterJSRAW">[0, 1]</code>.</p>
<h2>Concatenation of Dissimilar DataFrames Filled With NaN</h2>
<p><em><strong>Problem</strong>: How to concatenate two DataFrames if they have different columns?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv")
df2 = pd.read_csv("Cars2.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- # Additional Dataframe "df2"
# ----------
# make origin
# 0 skoda Czechia
# 1 toyota Japan
# 2 ford USA
# ---------- try: result = pd.concat([df, df2], axis=0, ignore_index=True) print("Y")
except Exception: print ("N") # Y</pre>
<p>Even if DataFrames have different columns, you can concatenate them.</p>
<p>If DataFrame 1 has columns A and B and DataFrame 2 has columns C and D, the result of concatenating DataFrames 1 and 2 is a DataFrame with columns A, B, C, and D. Missing values in the rows are filled with <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>.</p>
<h2>Outer Merge</h2>
<p><em><strong>Problem</strong>: When merging (=joining) two DataFrames—what happens if there are missing values?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv")
df2 = pd.read_csv("Cars2.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- # Additional dataframe "df2"
# ----------
# make origin
# 0 skoda Czechia
# 1 mazda Japan
# 2 ford USA
# ---------- result = pd.merge(df, df2, how="outer", left_on="make", right_on="make")
print(len(result["fuel"]))
print(result["fuel"].count())
# 7
# 5</pre>
<p>With Panda’s function <code data-enlighter-language="generic" class="EnlighterJSRAW">merge()</code> and the parameter <code data-enlighter-language="generic" class="EnlighterJSRAW">how</code> set to <code data-enlighter-language="generic" class="EnlighterJSRAW">outer</code>, you can perform an <strong>outer join</strong>.</p>
<p>The resulting DataFrame of an outer join contains all values from both input DataFrames; missing values are filled with <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>.</p>
<p>In addition, this puzzle shows how <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values are counted by the <code data-enlighter-language="generic" class="EnlighterJSRAW"><a href="https://blog.finxter.com/python-list-length-whats-the-runtime-complexity-of-len/" title="Python List Length – What’s the Runtime Complexity of len()?">len()</a></code> function whereas the method <code data-enlighter-language="generic" class="EnlighterJSRAW">count()</code> does not include <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values.</p>
<h2>Replacing NaN </h2>
<p><em><strong>Problem</strong>: How to Replace all <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values in a DataFrame with a given value?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- df.fillna(2.0, inplace=True)
result = df["engine-size"].sum()
print(result)
# 13.8</pre>
<p>The method <code data-enlighter-language="generic" class="EnlighterJSRAW">fillna()</code> replaces <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values with a new value. Thus, the sum of all values in the column <code data-enlighter-language="generic" class="EnlighterJSRAW">engine-size</code> is 13.8.</p>
<h2>Length vs. Count Difference — It’s NaN!</h2>
<p><em><strong>Problem</strong>: What’s the difference between the <code data-enlighter-language="generic" class="EnlighterJSRAW">len()</code> and the <code data-enlighter-language="generic" class="EnlighterJSRAW">count()</code> functions?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv")
df2 = pd.read_csv("Cars2.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- # Additional dataframe "df2"
# ----------
# make origin
# 0 skoda Czechia
# 1 mazda Japan
# 2 ford USA
# ---------- result = pd.merge(df2, df, how="left", left_on="make", right_on="make")
print(len(result["fuel"]))
print(result["fuel"].count())
# 3
# 1</pre>
<p>In a left join, the left DataFrame is the master, and all its values are included in the resulting DataFrame.</p>
<p>Therefore, the result DataFrame contains three rows, yet, since <code data-enlighter-language="generic" class="EnlighterJSRAW">skoda</code> and <code data-enlighter-language="generic" class="EnlighterJSRAW">ford</code> don’t appear in DataFrame <code data-enlighter-language="generic" class="EnlighterJSRAW">df</code>, only one the row for <code data-enlighter-language="generic" class="EnlighterJSRAW">mazda</code> contains value.</p>
<p>Again, we see the difference between using the function <code data-enlighter-language="generic" class="EnlighterJSRAW">len()</code> which also includes <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values and the method <code data-enlighter-language="generic" class="EnlighterJSRAW">count()</code> which does not count <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values.</p>
<h2>Equals() vs. == When Comparing NaN</h2>
<p>Problem: </p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- df["engine-size_copy"] = df["engine-size"]
check1 = (df["engine-size_copy"] == df["engine-size"]).all()
check2 = df["engine-size_copy"].equals(df["engine-size"])
print(check1 == check2)
# False</pre>
<p>This code snippet shows how to compare columns or entire DataFrames regarding the shape and the elements.</p>
<p>The comparison using the operator <code data-enlighter-language="generic" class="EnlighterJSRAW">==</code> returns <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code> for our DataFrame because the comparing <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-values with <code data-enlighter-language="generic" class="EnlighterJSRAW">==</code> always yields <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code>.</p>
<p>On the other hand, <code data-enlighter-language="generic" class="EnlighterJSRAW">df.equals()</code> allows comparing two Series or DataFrames. In this case, <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-values in the same location are considered to be equal.</p>
<p>The column headers do not need to have the same type, but the elements within the columns must be of the same <code data-enlighter-language="generic" class="EnlighterJSRAW">dtype</code>.</p>
<p>Since the result of <code data-enlighter-language="generic" class="EnlighterJSRAW">check1</code> is <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code> and the result of <code data-enlighter-language="generic" class="EnlighterJSRAW">check2</code> yields <code data-enlighter-language="generic" class="EnlighterJSRAW">True</code>, the final output is <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code>.</p>
<h2>Where to Go From Here?</h2>
<p>Enough theory, let’s get some practice!</p>
<p>To become successful in coding, you need to get out there and solve real problems for real people. That’s how you can become a six-figure earner easily. And that’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?</p>
<p><strong>Practice projects is how you sharpen your saw in coding!</strong></p>
<p>Do you want to become a code master by focusing on practical code projects that actually earn you money and solve problems for people?</p>
<p>Then become a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.</p>
<p>Join my free webinar <a rel="noreferrer noopener" href="https://blog.finxter.com/webinar-freelancer/" target="_blank">“How to Build Your High-Income Skill Python”</a> and watch how I grew my coding business online and how you can, too—from the comfort of your own home.</p>
<p><a href="https://blog.finxter.com/webinar-freelancer/" target="_blank" rel="noreferrer noopener">Join the free webinar now!</a></p>
<p>The post <a href="https://blog.finxter.com/pandas-nan/" target="_blank" rel="noopener noreferrer">Pandas NaN — Working With Missing Data</a> first appeared on <a href="https://blog.finxter.com/" target="_blank" rel="noopener noreferrer">Finxter</a>.</p>
</div>
https://www.sickgaming.net/blog/2020/11/...sing-data/
<div><p><em>Pandas is Excel on steroids—the powerful Python library allows you to analyze structured and tabular data with surprising efficiency and ease. Pandas is one of the reasons why master coders reach 100x the efficiency of average coders. In today’s article, you’ll learn how to work with missing data—in particular, how to <strong>handle NaN values</strong> in <a href="https://blog.finxter.com/pandas-cheat-sheets/" title="[PDF Collection] 7 Beautiful Pandas Cheat Sheets — Post Them to Your Wall" target="_blank" rel="noreferrer noopener">Pandas DataFrames</a>.</em></p>
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio">
<div class="wp-block-embed__wrapper">
<div class="ast-oembed-container"><iframe title="PANDAS ULTIMATE GUIDE TO NAN (NOT A NUMBER)" width="1400" height="788" src="https://www.youtube.com/embed/ihslLmCnlTw?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>
</div>
</figure>
<p>You’ll learn about all the different reasons why NaNs appear in your DataFrames—and how to handle them. Let’s get started!</p>
<h2>Checking Series for NaN Values</h2>
<p><em><strong>Problem</strong>: How to check a series for NaN values?</em></p>
<p>Have a look at the following code:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd
import numpy as np data = pd.Series([0, np.NaN, 2])
result = data.hasnans print(result)
# True</pre>
<p>Series can contain <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-values—an abbreviation for <em>Not-A-Number</em>—that describe undefined values.</p>
<p>To check if a Series contains one or more <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value, use the attribute <code data-enlighter-language="generic" class="EnlighterJSRAW">hasnans</code>. The attribute returns <code data-enlighter-language="generic" class="EnlighterJSRAW">True</code> if there is at least one <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value and <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code> otherwise. </p>
<p>There’s a <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value in the Series, so the output is <code data-enlighter-language="generic" class="EnlighterJSRAW">True</code>.</p>
<h2>Filtering Series Generates NaN</h2>
<p><em><strong>Problem</strong>: When <a href="https://blog.finxter.com/how-to-filter-a-list-in-python/" title="How to Filter a List in Python?" target="_blank" rel="noreferrer noopener">filtering </a>a Series with <code data-enlighter-language="generic" class="EnlighterJSRAW">where()</code> and no element passes the filtering condition, what’s the result?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd xs = pd.Series([5, 1, 4, 2, 3])
xs.where(xs > 2, inplace=True)
result = xs.hasnans print(result)
# True</pre>
<p>The method <code data-enlighter-language="generic" class="EnlighterJSRAW">where()</code> filters a Series by a <a href="https://blog.finxter.com/how-to-conditionally-select-elements-in-a-numpy-array/" title="How to Conditionally Select Elements in a Numpy Array?" target="_blank" rel="noreferrer noopener">condition</a>. Only the elements that satisfy the condition remain in the resulting Series. And what happens if a value doesn’t satisfy the condition? Per default, all rows not satisfying the condition are filled with <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-values.</p>
<p>This is why our Series contains <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-values after filtering it with the method <code data-enlighter-language="generic" class="EnlighterJSRAW">where()</code>.</p>
<h2>Working with Multiple Series of Different Lengths</h2>
<p><em><strong>Problem</strong>: If you element-wise add two Series objects with a different number of elements—what happens with the remaining elements?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd s = pd.Series(range(0, 10))
t = pd.Series(range(0, 20))
result = (s + t)[1] print(result)
# 2</pre>
<p>To add two Series element-wise, use the default addition operator <code data-enlighter-language="generic" class="EnlighterJSRAW">+</code>. The Series do not need to have the same size because once the first Series ends, the subsequent element-wise results are <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values.</p>
<p>At index <code data-enlighter-language="generic" class="EnlighterJSRAW">1</code> in the resulting Series, you get the result of <code data-enlighter-language="generic" class="EnlighterJSRAW">1 + 1 = 2</code>.</p>
<h2>Create a DataFrame From a List of Dictionaries with Unequal Keys</h2>
<p><em><strong>Problem</strong>: How to create a DataFrame from a list of dictionaries if the dictionaries have unequal keys? A DataFrame expects the same columns to be available for each row!</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd data = [{'Car':'Mercedes', 'Driver':'Hamilton, Lewis'}, {'Car':'Ferrari', 'Driver':'Schumacher, Michael'}, {'Car':'Lamborghini'}] df = pd.DataFrame(data, index=['Rank 2', 'Rank 1', 'Rank 3'])
df.sort_index(inplace=True)
result = df['Car'].iloc[0] print(result)
# Ferrari</pre>
<p>You can create a DataFrame from a <a href="https://blog.finxter.com/how-to-create-a-list-of-dictionaries-in-python/" target="_blank" rel="noreferrer noopener" title="How to Create a List of Dictionaries in Python?">list of dictionaries</a>. The dictionaries’ keys define the column labels, and the values define the columns’ entries. Not all dictionaries must contain the same keys. If a dictionary doesn’t contain a particular key, this will be interpreted as a <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-value.</p>
<p>This code snippet uses string labels as index values to <a href="https://blog.finxter.com/how-to-sort-a-dictionary-by-value-in-python/" title="How To Sort A Dictionary By Value in Python?" target="_blank" rel="noreferrer noopener">sort </a>the DataFrame. After sorting the DataFrame, the row with index label <code data-enlighter-language="generic" class="EnlighterJSRAW">Rank 1</code> is at location <code data-enlighter-language="generic" class="EnlighterJSRAW">0</code> in the DataFrame and the value in the column <code data-enlighter-language="generic" class="EnlighterJSRAW">Car</code> is <code data-enlighter-language="generic" class="EnlighterJSRAW">Ferrari</code>.</p>
<h2>Sorting a DataFrame by Column with NaN Values</h2>
<p><em><strong>Problem</strong>: What happens if you sort a DataFrame by column if the column contains a <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- selection = df.sort_values(by="engine-size")
result = selection.index.to_list()[0]
print(result)
# 1</pre>
<p>In this code snippet, you sort the rows of the DataFrame by the values of the column <code data-enlighter-language="generic" class="EnlighterJSRAW">engine-size</code>.</p>
<p>The main point is that <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values are always moved to the end in Pandas sorting. Thus, the first value is <code data-enlighter-language="generic" class="EnlighterJSRAW">1.8</code>, which belongs to the row with index value <code data-enlighter-language="generic" class="EnlighterJSRAW">1</code>.</p>
<h2>Count Non-NaN Values</h2>
<p><em><strong>Problem</strong>: How to count the number of elements in a dataframe column that are not <code data-enlighter-language="generic" class="EnlighterJSRAW">Nan</code>?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- df.count()[5]
print(result)
# 4</pre>
<p>The method <code data-enlighter-language="generic" class="EnlighterJSRAW">count()</code> returns the number of non-<code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values for each column. The DataFrame <code data-enlighter-language="generic" class="EnlighterJSRAW">df</code> has five rows. The fifth column<br />contains one <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value. Therefore, the count of the fifth column is <code data-enlighter-language="generic" class="EnlighterJSRAW">4</code>.</p>
<h2>Drop NaN-Values</h2>
<p><em><strong>Problem</strong>: How to drop all rows that contain a <code>NaN</code> value in any of its columns—and how to restrict this to certain columns?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- selection1 = df.dropna(subset=["price"])
selection2 = df.dropna()
print(len(selection1), len(selection2))
# 5 4</pre>
<p>The DataFrame’s <code data-enlighter-language="generic" class="EnlighterJSRAW">dropna()</code> method drops all rows that contain a <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value in any of its columns. But how to restrict the columns to be scanned for <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values?</p>
<p>By passing a list of column labels to the optional parameter <code data-enlighter-language="generic" class="EnlighterJSRAW">subset</code>, you can define which columns you want to consider.</p>
<p>The call of <code data-enlighter-language="generic" class="EnlighterJSRAW">dropna()</code> without restriction, drops line <code data-enlighter-language="generic" class="EnlighterJSRAW">2</code> because of the <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value in the column <code data-enlighter-language="generic" class="EnlighterJSRAW">engine-size</code>. When you restrict the columns only to <code data-enlighter-language="generic" class="EnlighterJSRAW">price</code>, no rows will be dropped, because no <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> value is present.</p>
<h2>Drop Nan and Reset Index</h2>
<p><em><strong>Problem</strong>: What happens to indices after dropping certain rows?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- df.drop([0, 1, 2], inplace=True)
df.reset_index(inplace=True)
result = df.index.to_list()
print(result)
# [0, 1]</pre>
<p>The method <code data-enlighter-language="generic" class="EnlighterJSRAW">drop()</code> on a DataFrame deletes rows or columns by index. You can either pass a single value or a list of values.</p>
<p>By default the <code data-enlighter-language="generic" class="EnlighterJSRAW">inplace</code> parameter is set to <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code>, so that modifications won’t affect the initial DataFrame object. Instead, the method returns a modified copy of the DataFrame. In the puzzle, you set <code data-enlighter-language="generic" class="EnlighterJSRAW">inplace</code> to <code data-enlighter-language="generic" class="EnlighterJSRAW">True</code>, so the deletions are performed directly on the DataFrame.</p>
<p>After deleting the first three rows, the first two index labels are 3 and 4. You can reset the default indexing by calling the method <code data-enlighter-language="generic" class="EnlighterJSRAW">reset_index()</code> on the DataFrame, so that the index starts at 0 again. As there are only two rows left in the DataFrame, the result is <code data-enlighter-language="generic" class="EnlighterJSRAW">[0, 1]</code>.</p>
<h2>Concatenation of Dissimilar DataFrames Filled With NaN</h2>
<p><em><strong>Problem</strong>: How to concatenate two DataFrames if they have different columns?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv")
df2 = pd.read_csv("Cars2.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- # Additional Dataframe "df2"
# ----------
# make origin
# 0 skoda Czechia
# 1 toyota Japan
# 2 ford USA
# ---------- try: result = pd.concat([df, df2], axis=0, ignore_index=True) print("Y")
except Exception: print ("N") # Y</pre>
<p>Even if DataFrames have different columns, you can concatenate them.</p>
<p>If DataFrame 1 has columns A and B and DataFrame 2 has columns C and D, the result of concatenating DataFrames 1 and 2 is a DataFrame with columns A, B, C, and D. Missing values in the rows are filled with <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>.</p>
<h2>Outer Merge</h2>
<p><em><strong>Problem</strong>: When merging (=joining) two DataFrames—what happens if there are missing values?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv")
df2 = pd.read_csv("Cars2.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- # Additional dataframe "df2"
# ----------
# make origin
# 0 skoda Czechia
# 1 mazda Japan
# 2 ford USA
# ---------- result = pd.merge(df, df2, how="outer", left_on="make", right_on="make")
print(len(result["fuel"]))
print(result["fuel"].count())
# 7
# 5</pre>
<p>With Panda’s function <code data-enlighter-language="generic" class="EnlighterJSRAW">merge()</code> and the parameter <code data-enlighter-language="generic" class="EnlighterJSRAW">how</code> set to <code data-enlighter-language="generic" class="EnlighterJSRAW">outer</code>, you can perform an <strong>outer join</strong>.</p>
<p>The resulting DataFrame of an outer join contains all values from both input DataFrames; missing values are filled with <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>.</p>
<p>In addition, this puzzle shows how <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values are counted by the <code data-enlighter-language="generic" class="EnlighterJSRAW"><a href="https://blog.finxter.com/python-list-length-whats-the-runtime-complexity-of-len/" title="Python List Length – What’s the Runtime Complexity of len()?">len()</a></code> function whereas the method <code data-enlighter-language="generic" class="EnlighterJSRAW">count()</code> does not include <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values.</p>
<h2>Replacing NaN </h2>
<p><em><strong>Problem</strong>: How to Replace all <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values in a DataFrame with a given value?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- df.fillna(2.0, inplace=True)
result = df["engine-size"].sum()
print(result)
# 13.8</pre>
<p>The method <code data-enlighter-language="generic" class="EnlighterJSRAW">fillna()</code> replaces <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values with a new value. Thus, the sum of all values in the column <code data-enlighter-language="generic" class="EnlighterJSRAW">engine-size</code> is 13.8.</p>
<h2>Length vs. Count Difference — It’s NaN!</h2>
<p><em><strong>Problem</strong>: What’s the difference between the <code data-enlighter-language="generic" class="EnlighterJSRAW">len()</code> and the <code data-enlighter-language="generic" class="EnlighterJSRAW">count()</code> functions?</em></p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv")
df2 = pd.read_csv("Cars2.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- # Additional dataframe "df2"
# ----------
# make origin
# 0 skoda Czechia
# 1 mazda Japan
# 2 ford USA
# ---------- result = pd.merge(df2, df, how="left", left_on="make", right_on="make")
print(len(result["fuel"]))
print(result["fuel"].count())
# 3
# 1</pre>
<p>In a left join, the left DataFrame is the master, and all its values are included in the resulting DataFrame.</p>
<p>Therefore, the result DataFrame contains three rows, yet, since <code data-enlighter-language="generic" class="EnlighterJSRAW">skoda</code> and <code data-enlighter-language="generic" class="EnlighterJSRAW">ford</code> don’t appear in DataFrame <code data-enlighter-language="generic" class="EnlighterJSRAW">df</code>, only one the row for <code data-enlighter-language="generic" class="EnlighterJSRAW">mazda</code> contains value.</p>
<p>Again, we see the difference between using the function <code data-enlighter-language="generic" class="EnlighterJSRAW">len()</code> which also includes <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values and the method <code data-enlighter-language="generic" class="EnlighterJSRAW">count()</code> which does not count <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code> values.</p>
<h2>Equals() vs. == When Comparing NaN</h2>
<p>Problem: </p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import pandas as pd df = pd.read_csv("Cars.csv") # Dataframe "df"
# ----------
# make fuel aspiration body-style price engine-size
# 0 audi gas turbo sedan 30000 2.0
# 1 dodge gas std sedan 17000 1.8
# 2 mazda diesel std sedan 17000 NaN
# 3 porsche gas turbo convertible 120000 6.0
# 4 volvo diesel std sedan 25000 2.0
# ---------- df["engine-size_copy"] = df["engine-size"]
check1 = (df["engine-size_copy"] == df["engine-size"]).all()
check2 = df["engine-size_copy"].equals(df["engine-size"])
print(check1 == check2)
# False</pre>
<p>This code snippet shows how to compare columns or entire DataFrames regarding the shape and the elements.</p>
<p>The comparison using the operator <code data-enlighter-language="generic" class="EnlighterJSRAW">==</code> returns <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code> for our DataFrame because the comparing <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-values with <code data-enlighter-language="generic" class="EnlighterJSRAW">==</code> always yields <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code>.</p>
<p>On the other hand, <code data-enlighter-language="generic" class="EnlighterJSRAW">df.equals()</code> allows comparing two Series or DataFrames. In this case, <code data-enlighter-language="generic" class="EnlighterJSRAW">NaN</code>-values in the same location are considered to be equal.</p>
<p>The column headers do not need to have the same type, but the elements within the columns must be of the same <code data-enlighter-language="generic" class="EnlighterJSRAW">dtype</code>.</p>
<p>Since the result of <code data-enlighter-language="generic" class="EnlighterJSRAW">check1</code> is <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code> and the result of <code data-enlighter-language="generic" class="EnlighterJSRAW">check2</code> yields <code data-enlighter-language="generic" class="EnlighterJSRAW">True</code>, the final output is <code data-enlighter-language="generic" class="EnlighterJSRAW">False</code>.</p>
<h2>Where to Go From Here?</h2>
<p>Enough theory, let’s get some practice!</p>
<p>To become successful in coding, you need to get out there and solve real problems for real people. That’s how you can become a six-figure earner easily. And that’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?</p>
<p><strong>Practice projects is how you sharpen your saw in coding!</strong></p>
<p>Do you want to become a code master by focusing on practical code projects that actually earn you money and solve problems for people?</p>
<p>Then become a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.</p>
<p>Join my free webinar <a rel="noreferrer noopener" href="https://blog.finxter.com/webinar-freelancer/" target="_blank">“How to Build Your High-Income Skill Python”</a> and watch how I grew my coding business online and how you can, too—from the comfort of your own home.</p>
<p><a href="https://blog.finxter.com/webinar-freelancer/" target="_blank" rel="noreferrer noopener">Join the free webinar now!</a></p>
<p>The post <a href="https://blog.finxter.com/pandas-nan/" target="_blank" rel="noopener noreferrer">Pandas NaN — Working With Missing Data</a> first appeared on <a href="https://blog.finxter.com/" target="_blank" rel="noopener noreferrer">Finxter</a>.</p>
</div>
https://www.sickgaming.net/blog/2020/11/...sing-data/