{"id":112725,"date":"2020-05-10T09:55:25","date_gmt":"2020-05-10T09:55:25","guid":{"rendered":"https:\/\/blog.finxter.com\/?p=8400"},"modified":"2020-05-10T09:55:25","modified_gmt":"2020-05-10T09:55:25","slug":"python-list-of-lists-group-by-a-simple-illustrated-guide","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2020\/05\/10\/python-list-of-lists-group-by-a-simple-illustrated-guide\/","title":{"rendered":"Python List of Lists Group By \u2013 A Simple Illustrated Guide"},"content":{"rendered":"<figure class=\"wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\">\n<div class=\"wp-block-embed__wrapper\">\n<div class=\"ast-oembed-container\"><iframe loading=\"lazy\" title=\"Python List of Lists Group By \u2013 A Simple Illustrated Guide\" width=\"1400\" height=\"788\" src=\"https:\/\/www.youtube.com\/embed\/NaItYmuwPcs?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/div>\n<\/div>\n<\/figure>\n<p>This tutorial shows you how to group the inner <a rel=\"noreferrer noopener\" href=\"https:\/\/blog.finxter.com\/python-lists\/\" target=\"_blank\">lists <\/a>of a Python <a href=\"https:\/\/blog.finxter.com\/python-list-of-lists\/\">list of lists<\/a> by common element. There are three basic methods: <\/p>\n<ol>\n<li>Group the inner lists together by common element.<\/li>\n<li>Group the inner lists together by common element AND aggregating them (e.g. averaging).<\/li>\n<li>Group the inner lists together by common element AND aggregating them (e.g. averaging) using the <a href=\"https:\/\/pandas.pydata.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">Pandas<\/a> external library.<\/li>\n<\/ol>\n<p>Before we explore these three options in more detail, let&#8217;s give you the quick solution first using the Pandas library in our interactive shell:<\/p>\n<figure><iframe loading=\"lazy\" src=\"https:\/\/repl.it\/@finxter\/pandaslistoflistsgroupby?lite=true\" allowfullscreen=\"true\" width=\"100%\" height=\"1000px\"><\/iframe><\/figure>\n<p>You can run this code in your browser. If you want to learn about the <a href=\"https:\/\/blog.finxter.com\/python-crash-course\/\" target=\"_blank\" rel=\"noreferrer noopener\">Pythonic <\/a>alternatives or you need a few more explanations, then read on!<\/p>\n<h2>Method 1: Group List of Lists By Common Element in Dictionary<\/h2>\n<p><strong>Problem<\/strong>: Given a <a href=\"https:\/\/blog.finxter.com\/python-list-of-lists\/\" target=\"_blank\" rel=\"noreferrer noopener\">list of lists<\/a>. Group the elements by common element and store the result in a <a href=\"https:\/\/blog.finxter.com\/python-dictionary\/\" target=\"_blank\" rel=\"noreferrer noopener\">dictionary <\/a>(key = common element). <\/p>\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/05\/groupby-1024x576.jpg\" alt=\"\" class=\"wp-image-8405\" width=\"768\" height=\"432\" srcset=\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/05\/groupby-scaled.jpg 1024w, https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/05\/groupby-300x169.jpg 300w, https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/05\/groupby-768x432.jpg 768w\" sizes=\"auto, (max-width: 768px) 100vw, 768px\" \/><\/figure>\n<p><strong>Example<\/strong>: Say, you&#8217;ve got a database with multiple rows (the list of lists) where each row consists of three attributes: Name, Age, and Income. You want to group by Name and store the result in a dictionary. The dictionary keys are given by the Name attribute. The dictionary values are a list of rows that have this exact Name attribute.<\/p>\n<p><strong>Solution<\/strong>: Here&#8217;s the data and how you can group by a common attribute (e.g., Name).<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"># Database:\n# row = [Name, Age, Income]\nrows = [['Alice', 19, 45000], ['Bob', 18, 22000], ['Ann', 26, 88000], ['Alice', 33, 118000]] # Create a dictionary grouped by Name\nd = {}\nfor row in rows: # Add name to dict if not exists if row[0] not in d: d[row[0]] = [] # Add all non-Name attributes as a new list d[row[0]].append(row[1:]) print(d)\n# {'Alice': [[19, 45000], [33, 118000]],\n# 'Bob': [[18, 22000]],\n# 'Ann': [[26, 88000]]}<\/pre>\n<p>You can see that the result is a dictionary with one key per name (<code>'Alice'<\/code>, <code>'Bob'<\/code>, and <code>'Ann'<\/code>). Alice appears in two rows of the original database (list of lists). Thus, you associate two rows to her name&#8212;maintaining only the Age and Income attributes per row. <\/p>\n<p>The strategy how you accomplish this is simple: <\/p>\n<ul>\n<li>Create the empty dictionary.<\/li>\n<li>Go over each row in the list of lists. The first value of the row list is the Name attribute.<\/li>\n<li>Add the Name attribute <code>row[0]<\/code> to the dictionary if it doesn&#8217;t exist, yet&#8212;initializing the dictionary to the empty list. Now, you can be sure that the key exist in the dictionary.<\/li>\n<li>Append the <a rel=\"noreferrer noopener\" href=\"https:\/\/blog.finxter.com\/introduction-to-slicing-in-python\/\" target=\"_blank\">sublist slic<\/a><a href=\"https:\/\/blog.finxter.com\/introduction-to-slicing-in-python\/\" target=\"_blank\" rel=\"noreferrer noopener\">e<\/a><a rel=\"noreferrer noopener\" href=\"https:\/\/blog.finxter.com\/introduction-to-slicing-in-python\/\" target=\"_blank\"> <\/a><code>[Age, Income]<\/code> to the dictionary value so that this becomes a list of lists as well&#8212;one list per database row.<\/li>\n<li>You&#8217;ve now grouped all database entries by a common attribute (=Name).<\/li>\n<\/ul>\n<p>So far, so good. But what if you want to perform some aggregation on the grouped database rows?<\/p>\n<h2>Method 2: Group List of Lists By Common Element and Aggregate Grouped Elements<\/h2>\n<p><strong>Problem<\/strong>: In the previous example, you&#8217;ve seen that each dictionary value is a list of lists because you store each row as a separate list. But what if you want to aggregate all grouped rows? <\/p>\n<p><strong>Example<\/strong>: The dictionary entry for the key <code>'Alice'<\/code> may be <code>[[19, 45000], [33, 118000]]<\/code> but you want to <a href=\"https:\/\/blog.finxter.com\/how-to-average-a-list-of-lists-in-python\/\" target=\"_blank\" rel=\"noreferrer noopener\">average <\/a>the age and income values: <code>[(19+33)\/2, (45000+118000)\/2]<\/code>. How do you do that?<\/p>\n<p><strong>Solution<\/strong>: The solution is simply to add one post-processing step after the above code to aggregate all attributes using the <a rel=\"noreferrer noopener\" href=\"https:\/\/blog.finxter.com\/zip-unzip-python\/\" target=\"_blank\"><code>zip()<\/code> function<\/a> as follows. Note that this is the exact same code as before (without aggregation) with three lines added at the end to aggregate the list of lists for each grouped Name into a single <a href=\"https:\/\/blog.finxter.com\/python-list-average\/\" target=\"_blank\" rel=\"noreferrer noopener\">average<\/a> value.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"># Database:\n# row = [Name, Age, Income]\nrows = [['Alice', 19, 45000], ['Bob', 18, 22000], ['Ann', 26, 88000], ['Alice', 33, 118000]] # Create a dictionary grouped by Name\nd = {}\nfor row in rows: # Add name to dict if not exists if row[0] not in d: d[row[0]] = [] # Add all non-Name attributes as a new list d[row[0]].append(row[1:]) print(d)\n# {'Alice': [[19, 45000], [33, 118000]],\n# 'Bob': [[18, 22000]],\n# 'Ann': [[26, 88000]]} # AGGREGATION FUNCTION:\nfor key in d: d[key] = [sum(x) \/ len(x) for x in zip(*d[key])] print(d)\n# {'Alice': [26.0, 81500.0], 'Bob': [18.0, 22000.0], 'Ann': [26.0, 88000.0]}\n<\/pre>\n<p>In the code, you use the aggregation function <code>sum(x) \/ len(x)<\/code> to calculate the average value for each attribute of the grouped rows. But you can replace this part with your own aggregation function such as <a href=\"https:\/\/blog.finxter.com\/how-to-calculate-weighted-average-numpy-array-along-axis\/\" target=\"_blank\" rel=\"noreferrer noopener\">average<\/a>, <a href=\"https:\/\/blog.finxter.com\/how-to-get-the-variance-of-a-list-in-python\/\" target=\"_blank\" rel=\"noreferrer noopener\">variance<\/a>, <a href=\"https:\/\/blog.finxter.com\/python-list-length-whats-the-runtime-complexity-of-len\/\" target=\"_blank\" rel=\"noreferrer noopener\">length<\/a>, <a href=\"https:\/\/blog.finxter.com\/how-to-get-the-key-with-minimum-value-in-a-python-dictionary\/\" target=\"_blank\" rel=\"noreferrer noopener\">minimum<\/a>, <a href=\"https:\/\/blog.finxter.com\/how-to-get-the-key-with-the-maximum-value-in-a-dictionary\/\" target=\"_blank\" rel=\"noreferrer noopener\">maximum<\/a>, etc.<\/p>\n<p><strong>Explanation<\/strong>: <\/p>\n<ul>\n<li>You go over each key in the <a href=\"https:\/\/blog.finxter.com\/python-dictionary\/\" target=\"_blank\" rel=\"noreferrer noopener\">dictionary <\/a>(the Name attribute) and aggregate the list of lists into a flat list of averaged attributes.<\/li>\n<li>You zip the attributes together. For example, <code>zip(*d['Alice'])<\/code> becomes <code>[[19, 33], [45000, 118000]]<\/code> (conceptually). <\/li>\n<li>You iterate over each list <code>x<\/code> of this list of lists in the <a rel=\"noreferrer noopener\" href=\"https:\/\/blog.finxter.com\/list-comprehension\/\" target=\"_blank\">list comprehension<\/a> statement.<\/li>\n<li>You aggregate the grouped attributes using your own custom function (e.g. <code>sum(x) \/ len(x)<\/code> to average the attribute values). <\/li>\n<\/ul>\n<p>See what happens in this code snippet in this interactive memory visualization tool (by clicking &#8220;Next&#8221;):<\/p>\n<p> <iframe loading=\"lazy\" width=\"800\" height=\"500\" frameborder=\"0\" src=\"https:\/\/pythontutor.com\/iframe-embed.html#code=%23%20Database%3A%0A%23%20row%20%3D%20%5BName,%20Age,%20Income%5D%0Arows%20%3D%20%5B%5B'Alice',%2019,%2045000%5D,%0A%20%20%20%20%20%20%20%20%5B'Bob',%2018,%2022000%5D,%0A%20%20%20%20%20%20%20%20%5B'Ann',%2026,%2088000%5D,%0A%20%20%20%20%20%20%20%20%5B'Alice',%2033,%20118000%5D%5D%0A%0A%0A%23%20Create%20a%20dictionary%20grouped%20by%20Name%0Ad%20%3D%20%7B%7D%0Afor%20row%20in%20rows%3A%0A%0A%20%20%20%20%23%20Add%20name%20to%20dict%20if%20not%20exists%0A%20%20%20%20if%20row%5B0%5D%20not%20in%20d%3A%0A%20%20%20%20%20%20%20%20d%5Brow%5B0%5D%5D%20%3D%20%5B%5D%0A%0A%20%20%20%20%23%20Add%20all%20non-Name%20attributes%20as%20a%20new%20list%0A%20%20%20%20d%5Brow%5B0%5D%5D.append%28row%5B1%3A%5D%29%0A%0A%23%20AGGREGATION%20FUNCTION%3A%0Afor%20key%20in%20d%3A%0A%20%20%20%20d%5Bkey%5D%20%3D%20%5Bsum%28x%29%20\/%20len%28x%29%20for%20x%20in%20zip%28*d%5Bkey%5D%29%5D%0A%0Aprint%28d%29%0A&#038;codeDivHeight=400&#038;codeDivWidth=350&#038;cumulative=false&#038;curInstr=0&#038;heapPrimitives=nevernest&#038;origin=opt-frontend.js&#038;py=3&#038;rawInputLstJSON=%5B%5D&#038;textReferences=false\"> <\/iframe> <\/p>\n<h2>Method 3: Pandas GroupBy<\/h2>\n<p>The <a href=\"https:\/\/blog.finxter.com\/pandas-cheat-sheets\/\" target=\"_blank\" rel=\"noreferrer noopener\">Pandas library<\/a> has its own powerful implementation of the <a rel=\"noreferrer noopener\" href=\"https:\/\/pandas.pydata.org\/pandas-docs\/stable\/reference\/api\/pandas.DataFrame.groupby.html\" target=\"_blank\">groupby() function<\/a>. Have a look at the code first:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"># Database:\n# row = [Name, Age, Income]\nrows = [['Alice', 19, 45000], ['Bob', 18, 22000], ['Ann', 26, 88000], ['Alice', 33, 118000]] import pandas as pd\ndf = pd.DataFrame(rows) print(df) ''' 0 1 2\n0 Alice 19 45000\n1 Bob 18 22000\n2 Ann 26 88000\n3 Alice 33 118000 ''' print(df.groupby([0]).mean()) ''' 1 2\n0 Alice 26 81500\nAnn 26 88000\nBob 18 22000 '''<\/pre>\n<p><strong>Explanation<\/strong>:<\/p>\n<ul>\n<li>Import the pandas library. <a href=\"https:\/\/blog.finxter.com\/pandas-cheat-sheets\/\" target=\"_blank\" rel=\"noreferrer noopener\">Find your quick refresher cheat sheets here.<\/a><\/li>\n<li>Create a DataFrame object from the rows&#8212;think of it as an Excel spreadsheet in your code (with numbered rows and columns). <\/li>\n<li>Call the <code>groupby()<\/code> function on your DataFrame. Use the column index <code>[0]<\/code> (which is the Name attribute) to group your data. This creates a <code>DataFrameGroupBy<\/code> object.<\/li>\n<li>On the <code>DataFrameGroupBy<\/code> object call the <code>mean()<\/code> function or any other aggregator function you want. <\/li>\n<li>The result is the &#8220;spreadsheet&#8221; with grouped Name attributes where multiple rows with the same Name attributes are averaged (element-wise).<\/li>\n<\/ul>\n<h2>Where to Go From Here?<\/h2>\n<p>Enough theory, let\u2019s get some practice!<\/p>\n<p>To become successful in coding, you need to get out there and solve real problems for real people. That\u2019s how you can become a six-figure earner easily. And that\u2019s how you polish the skills you really need in practice. After all, what\u2019s the use of learning theory that nobody ever needs?<\/p>\n<p><strong>Practice projects is how you sharpen your saw in coding!<\/strong><\/p>\n<p>Do you want to become a code master by focusing on practical code projects that actually earn you money and solve problems for people?<\/p>\n<p>Then become a Python freelance developer! It\u2019s the best way of approaching the task of improving your Python skills\u2014even if you are a complete beginner.<\/p>\n<p>Join my free webinar <a rel=\"noreferrer noopener\" href=\"https:\/\/blog.finxter.com\/webinar-freelancer\/\" target=\"_blank\">\u201cHow to Build Your High-Income Skill Python\u201d<\/a> and watch how I grew my coding business online and how you can, too\u2014from the comfort of your own home.<\/p>\n<p><a href=\"https:\/\/blog.finxter.com\/webinar-freelancer\/\" target=\"_blank\" rel=\"noreferrer noopener\">Join the free webinar now!<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This tutorial shows you how to group the inner lists of a Python list of lists by common element. There are three basic methods: Group the inner lists together by common element. Group the inner lists together by common element AND aggregating them (e.g. averaging). Group the inner lists together by common element AND aggregating [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[857],"tags":[73,468,528],"class_list":["post-112725","post","type-post","status-publish","format-standard","hentry","category-python-tut","tag-programming","tag-python","tag-tutorial"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/112725","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=112725"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/112725\/revisions"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=112725"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=112725"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=112725"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}