{"id":118708,"date":"2020-09-30T08:21:52","date_gmt":"2020-09-30T08:21:52","guid":{"rendered":"https:\/\/blog.finxter.com\/?p=13975"},"modified":"2020-09-30T08:21:52","modified_gmt":"2020-09-30T08:21:52","slug":"how-to-remove-duplicates-from-a-python-list-while-preserving-order","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2020\/09\/30\/how-to-remove-duplicates-from-a-python-list-while-preserving-order\/","title":{"rendered":"How to Remove Duplicates From a Python List While Preserving Order?"},"content":{"rendered":"<p class=\"has-pale-cyan-blue-background-color has-background\">To remove duplicates from a Python list while preserving the order of the elements, use the code <code>list(dict.fromkeys(list))<\/code> that goes through two phases: (1) Convert the list to a dict using the <code>dict.fromkeys()<\/code> function with the list elements as keys and <code>None<\/code> as dict values. (2) Convert the dictionary back to a list using the <code>list()<\/code> constructor. As dictionaries preserve the order of the keys, the list ordering is preserved.<\/p>\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\">\n<div class=\"wp-block-embed__wrapper\">\n<div class=\"ast-oembed-container\"><iframe loading=\"lazy\" title=\"How to Remove Duplicates From a Python List?\" width=\"1400\" height=\"788\" src=\"https:\/\/www.youtube.com\/embed\/GXL23jfNk1Y?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/div>\n<\/div>\n<\/figure>\n<p><strong>Problem<\/strong>: How to remove duplicates from a <a href=\"https:\/\/blog.finxter.com\/python-lists\/\" title=\"The Ultimate Guide to Python Lists\" target=\"_blank\" rel=\"noreferrer noopener\">Python list<\/a> while keeping the order of the list elements preserved?<\/p>\n<p>You may find this question a little awkward. What has <a href=\"https:\/\/blog.finxter.com\/python-list-remove-duplicates-and-keep-the-order\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"Python List: Remove Duplicates and Keep the Order\">removing duplicates <\/a>to do with preserving the order of the elements? The reason is simple: a well-known and efficient way to remove duplicates from a list is to <a href=\"https:\/\/blog.finxter.com\/python-list-to-set\/\" title=\"Python List to Set Conversion [Interactive Guide]\" target=\"_blank\" rel=\"noreferrer noopener\">convert the list to a set<\/a>&#8212;which is duplicated-free&#8212;and <a href=\"https:\/\/blog.finxter.com\/python-set-to-list\/\" title=\"Python Convert Set to List [Interactive Guide]\" target=\"_blank\" rel=\"noreferrer noopener\">converting it back to a list<\/a>. Here&#8217;s what you may find everywhere:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">lst = [42, 42, 'Alice', 'Alice', 1]\ndup_free = list(set(lst))\nprint(dup_free)\n# ['Alice', 42, 1]\n<\/pre>\n<p>The back-and-forth conversion <code>list(set(lst))<\/code> removes all duplicates from the list. However, it doesn&#8217;t preserve the order of the elements. In the example, the string <code>'Alice'<\/code> now appears before the integer <code>42<\/code>. <\/p>\n<p><strong><em>So, how to remove duplicates while preserving the order of the elements?<\/em><\/strong><\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter is-resized\"><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/04\/removeDupsPython-1024x576.jpg\" alt=\"Remove Duplicates From List Preserve Order Python\" width=\"768\" height=\"432\"\/><\/figure>\n<\/div>\n<p>The most Pythonic and blazingly fast approach is to use a <a href=\"https:\/\/blog.finxter.com\/python-dictionary\/\" target=\"_blank\" rel=\"noreferrer noopener\">dictionary<\/a>:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">lst = [3, 3, 22, 22, 1]\nresult = list(dict.fromkeys(lst))\nprint(result)\n# [3, 22, 1]<\/pre>\n<p>The <code>dict.fromkeys()<\/code> method creates a new dictionary using the elements from an iterable as the keys.\u00a0Python dictionary keys are unique by default so <a href=\"https:\/\/blog.finxter.com\/list-to-dict-convert-a-list-into-a-dictionary-in-python\/\" title=\"List to Dict \u2014 Convert a List Into a Dictionary in Python\" target=\"_blank\" rel=\"noreferrer noopener\">converting our list into a dictionary<\/a> will remove duplicates automatically. Once this has been done with our initial list, converting the dictionary back results in the duplicate-free list.<\/p>\n<p>This is the most Pythonic way to remove duplicates from a Python list while preserving the order. <\/p>\n<p><strong>Is this method fast? <\/strong>Like sets, dictionaries use <a href=\"https:\/\/blog.finxter.com\/python-dictionary\/#What_is_Hashing_in_Python\" target=\"_blank\" rel=\"noreferrer noopener\">hash tables<\/a>, which means they are extremely fast.<\/p>\n<p class=\"wp-block-block\">Do you want to develop the skills of a <strong>well-rounded Python professional<\/strong>&#8212;while getting paid in the process? Become a Python freelancer and order your book <a href=\"https:\/\/amzn.to\/2Re2JqO\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Leaving the Rat Race with Python<\/strong><\/a> on Amazon (<em>Kindle\/Print<\/em>)!<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-medium is-resized\"><a href=\"https:\/\/amzn.to\/2Re2JqO\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/08\/final_cover-200x300.jpg\" alt=\"Leaving the Rat Race with Python Book\" class=\"wp-image-11850\" width=\"200\" height=\"300\" srcset=\"https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/08\/final_cover-200x300.jpg 200w, https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/08\/final_cover-scaled.jpg 683w, https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/08\/final_cover-768x1152.jpg 768w, https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/08\/final_cover-1024x1536.jpg 1024w, https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/08\/final_cover-1365x2048.jpg 1365w, https:\/\/blog.finxter.com\/wp-content\/uploads\/2020\/08\/final_cover-150x225.jpg 150w\" sizes=\"auto, (max-width: 200px) 100vw, 200px\" \/><\/a><\/figure>\n<\/div>\n<h2>Do Python Dictionaries Preserve the Ordering of the Keys?<\/h2>\n<p><strong>Surprisingly, <em>the dictionary keys in Python preserve the order of the elements<\/em>. So, yes, the order of the elements is preserved. <a rel=\"noreferrer noopener\" href=\"https:\/\/morepypy.blogspot.com\/2015\/01\/faster-more-memory-efficient-and-more.html\" target=\"_blank\">(source)<\/a><\/strong><\/p>\n<p>Countless online resources like <a rel=\"noreferrer noopener\" href=\"https:\/\/stackoverflow.com\/questions\/5629023\/the-order-of-keys-in-dictionaries\" target=\"_blank\">this<\/a> argue that the order of dictionary keys is not preserved. They assume that the underlying implementation of the dictionary key iterables uses <a href=\"https:\/\/blog.finxter.com\/sets-in-python\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"The Ultimate Guide to Python Sets \u2013 with Harry Potter Examples\">sets<\/a>\u2014and sets are well-known to be agnostic to the ordering of elements. But this assumption is wrong. The built-in Python dictionary implementation in cPython preserves the order.<\/p>\n<p>Here\u2019s another example:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">lst = ['Alice', 'Bob', 'Bob', 1, 1, 1, 2, 3, 3]\ndic = dict.fromkeys(lst)\nprint(dic)\n# {'Alice': None, 'Bob': None, 1: None, 2: None, 3: None}<\/pre>\n<p>You see that the order of elements is preserved so when converting it back, the original ordering of the list elements is still preserved: <\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">print(list(dic))\n# ['Alice', 'Bob', 1, 2, 3]<\/pre>\n<p>However, you cannot rely on it because any Python implementation could, theoretically, decide not to preserve the order (notice the \u201cCOULD\u201d here is 100% theoretical and does not apply to the default cPython implementation).<\/p>\n<p>If you need to be certain that the order is preserved, you can use the <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.python.org\/3\/library\/collections.html#collections.OrderedDict\" target=\"_blank\">ordered dictionary library<\/a>. In cPython, this is just a wrapper for the default dict implementation.<\/p>\n<p><strong>Source Article:<\/strong> <a href=\"https:\/\/blog.finxter.com\/how-to-remove-duplicates-from-a-python-list\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"https:\/\/blog.finxter.com\/how-to-remove-duplicates-from-a-python-list\/\">How to Remove Duplicates From a Python List?<\/a><\/p>\n<h2>Removing Duplicates From Ordered Lists For Older Versions<\/h2>\n<p>Dictionaries only became ordered in all Python implementations when <a href=\"https:\/\/blog.finxter.com\/how-to-check-your-python-version\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"How to Check Your Python Version? A Helpful Guide\">Python 3.7<\/a> was released (this was also an implementation detail of CPython 3.6).\u00a0<\/p>\n<p>So, if you\u2019re using an older version of Python, you will need to import the <code>OrderedDict<\/code> class from the collections package in the standard library instead:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"> from collections import OrderedDict lst = [1, 1, 9, 1, 9, 6, 9, 7] result = list(OrderedDict.fromkeys(lst))<\/pre>\n<p>The output is the following duplicate-free list with the order of the elements preserved:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"> print(result) # [1, 9, 6, 7]<\/pre>\n<h2>Interactive Code Shell<\/h2>\n<p>Let&#8217;s try this method in our interactive Python shell:<\/p>\n<figure><iframe loading=\"lazy\" src=\"https:\/\/repl.it\/repls\/GrandioseRemarkableBackups?lite=true\" allowfullscreen=\"true\" width=\"100%\" height=\"400px\"><\/iframe><\/figure>\n<p><strong><em>Exercise: <\/em><\/strong><em>Run the code. Does it work?<\/em><\/p>\n<p>You can find more ways to remove duplicates while preserving the order in this detailed blog article:<\/p>\n<p><strong>Related tutorial:<\/strong> <a href=\"https:\/\/blog.finxter.com\/python-list-remove-duplicates-and-keep-the-order\/\" target=\"_blank\" rel=\"noreferrer noopener\" title=\"Python List: Remove Duplicates and Keep the Order\">Python List: Remove Duplicates and Keep the Order<\/a><\/p>\n<h2 class=\"wp-block-block\">Where to Go From Here?<\/h2>\n<p>Enough theory, let\u2019s get some practice!<\/p>\n<p>To become successful in coding, you need to get out there and solve real problems for real people. That\u2019s how you can become a six-figure earner easily. And that\u2019s how you polish the skills you really need in practice. After all, what\u2019s the use of learning theory that nobody ever needs?<\/p>\n<p><strong>Practice projects is how you sharpen your saw in coding!<\/strong><\/p>\n<p>Do you want to become a code master by focusing on practical code projects that actually earn you money and solve problems for people?<\/p>\n<p>Then become a Python freelance developer! It\u2019s the best way of approaching the task of improving your Python skills\u2014even if you are a complete beginner.<\/p>\n<p>Join my free webinar <a rel=\"noreferrer noopener\" href=\"https:\/\/blog.finxter.com\/webinar-freelancer\/\" target=\"_blank\">\u201cHow to Build Your High-Income Skill Python\u201d<\/a> and watch how I grew my coding business online and how you can, too\u2014from the comfort of your own home.<\/p>\n<p><a href=\"https:\/\/blog.finxter.com\/webinar-freelancer\/\" target=\"_blank\" rel=\"noreferrer noopener\">Join the free webinar now!<\/a><\/p>\n<p>The post <a href=\"https:\/\/blog.finxter.com\/how-to-remove-duplicates-from-a-python-list-while-preserving-order\/\" target=\"_blank\" rel=\"noopener noreferrer\">How to Remove Duplicates From a Python List While Preserving Order?<\/a> first appeared on <a href=\"https:\/\/blog.finxter.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">Finxter<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>To remove duplicates from a Python list while preserving the order of the elements, use the code list(dict.fromkeys(list)) that goes through two phases: (1) Convert the list to a dict using the dict.fromkeys() function with the list elements as keys and None as dict values. (2) Convert the dictionary back to a list using the [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[857],"tags":[73,468,528],"class_list":["post-118708","post","type-post","status-publish","format-standard","hentry","category-python-tut","tag-programming","tag-python","tag-tutorial"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/118708","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=118708"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/118708\/revisions"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=118708"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=118708"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=118708"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}