[Tut] 5 Best Ways to Check a List for Duplicates in Python

[xSicKxBot]

5 Best Ways to Check a List for Duplicates in Python

Rate this post

Problem Formulation and Solution Overview

In this article, you’ll learn how to check a List for Duplicates in Python.

To make it more fun, we have the following running scenario:

The Finxter Academy has given you an extensive list of usernames. Somewhere along the line, duplicate entries were added. They need you to check if their [i]List contains duplicates. For testing purposes, a small sampling of this List is used.[/i]

Question: How would we write Python code to check a List for duplicate elements?

We can accomplish this task by one of the following options:

• Method 1: Use
  Code:

  set()
  and
  Code:

  List
  to return a Duplicate-Free List
  
  
• Method 2: Use
  Code:

  set()
  ,
  Code:

  For
  loop and
  Code:

  List
  to return a List of Duplicates found.
  
  
• Method 3: Use a
  Code:

  For
  loop to return Duplicates and Counts
  
  
• Method 4: Use
  Code:

  any()
  to check for Duplicates and return a Boolean
  
  
• Method 5: Use List Comprehension to return a List of all Duplicates
  

Method 1: Use set() and List to return a Duplicate-Free List

This method uses

Code:

set()

which removes any duplicate values (

Code:

set(users)

) to produce a Duplicate-Free

Code:

set()

. This set is then converted to a List (

Code:

list(set(users))

).

Code:

users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] dup_free = list(set(users)) print(dup_free)

This code declares a small sampling of Finxter usernames and saves them to

Code:

users

.

Next,

Code:

set()

is called and

Code:

users

is passed as an argument to the same. Then, the new set is converted to a

Code:

List

and saved to

Code:

dup_free

.



If

Code:

dup_free

was output to the terminal before converting to a

Code:

List

, the result would be a

Code:

set()

, which is not subscriptable. Meaning the elements are inaccessible in this format.

Output

Code:

{'csealker', 'cdriver', 'shoeguy', 'ollie3', 'kyliek', 'stewieboy', 'AmyP'}

Note: Any attempt to access an element from a set will result in a not subscriptable error.

In this example, the

Code:

set()

was converted to a

Code:

List

, and displays a

Code:

List

of Duplicate-Free values.

Output

Code:

['csealker', 'cdriver', 'shoeguy', 'ollie3', 'kyliek', 'stewieboy', 'AmyP']

Note: An empty set will result if no argument is passed.

Method 2: Use set(), For loop, and List to return a List of Duplicates Found

This method uses

Code:

set()

, and a

Code:

For

loop to check for and return any Duplicates found

Code:

(set(x for x in users if ((x in tmp) or tmp.add(x)))

) to

Code:

dups

. The

Code:

set()

is then converted to a

Code:

List

(

Code:

print(list(dups))

).

Here’s an example:

Code:

users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] tmp = set() dups = set(x for x in users if (x in tmp or tmp.add(x))) print(list(dups))

This code declares a small sampling of Finxter usernames and saves them to

Code:

users

.

Next, a new empty set,

Code:

tmp

is declared. A

Code:

For

loop is then instantiated to check each element in

Code:

users

for duplicates. If a duplicate is found, it is appended to

Code:

tmp

. The results save to

Code:

dups

as a

Code:

set()

.

Output

In this example, the

Code:

set()

was converted to a

Code:

List

and displays a

Code:

List

of Duplicates values found in the original

Code:

List

,

Code:

users

.

Code:

['kyliek', 'ollie3', 'shoeguy']

Method 3: Use a

Code:

For

loop to return [b]Duplicates and Counts[/b]

This method uses a

Code:

For

loop to navigate through and check each element of

Code:

users

while keeping track of all usernames and the number of times they appear. A Dictionary of Duplicates, including the Usernames and Counts returns.

Here’s an example:

Code:

count = {} dup_count = {} for i in users: if i not in count: count[i] = 1 else: count[i] += 1 dup_count[i] = count[i] print(dup_count)

This code declares two (2) empty sets,

Code:

count

and

Code:

dup_count

respectively.

A

Code:

For

loop is instantiated to loop through each element of

Code:

users

and does the following:

• If the element
  Code:

  i
  is not in
  Code:

  count
  , then the
  Code:

  count
  element (
  Code:

  count[i]=1
  ) is set to one (1).
  
  
• If element
  Code:

  i
  is found in
  Code:

  count
  , it falls to
  Code:

  else
  where one (1) is added to
  Code:

  count
  (
  Code:

  count[i]+=1
  ) and then added to
  Code:

  dup_count
  (
  Code:

  dup_count[i]=count[i]
  )
  

This code repeats until the end of

Code:

users

has been reached.

At this point, a Dictionary containing the Duplicates, and the number of times they appear displays.

Output

Code:

{'ollie3': 2, 'shoeguy': 2, 'kyliek': 2}

Method 4: Use Any to Check for Duplicate Values

This example uses

Code:

any()

, and passes the iterable

Code:

users

to iterate and locate Duplicates. If found,

Code:

True

returns. Otherwise,

Code:

False

returns. Best used on small Lists.

Code:

users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] dups = any(users.count(x) > 1 for x in users) print(dups)

This code declares a small sampling of Finxter usernames and saves them to

Code:

users

.

Next,

Code:

any()

is called and loops through each element of

Code:

users

checking to see if the element is a duplicate. If found,

Code:

True

is assigned. Otherwise,

Code:

False

is assigned. The result saves to

Code:

dups

and the output displays as follows:

Output

True



Method 5: Use List Comprehension to return a [b]List of all Duplicates[/b]

This method uses List Comprehension to loop through

Code:

users

, checking for duplicates. If found, the Duplicates are appended to

Code:

dups

.

Here’s an example:

Code:

users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] dups = [x for x in users if users.count(x) >= 2] print(dups)

This code declares a small sampling of Finxter usernames and saves them to

Code:

users

.

Next, List Comprehension extracts and displays duplicate usernames and save them to a List. The duplicate values are output to the terminal

Output

Code:

['ollie3', 'shoeguy', 'kyliek', 'ollie3', 'shoeguy', 'kyliek']

Summary

These five (5) methods of checking a List for Duplicates should give you enough information to select the best one for your coding requirements.

Good Luck & Happy Coding!