[Tut] 5 Best Ways to Check a List for Duplicates in Python

[Tut] 5 Best Ways to Check a List for Duplicates in Python - Printable Version

+- Sick Gaming (https://sickgaming.net)
+-- Forum: Programming (https://sickgaming.net/forum-76.html)
+--- Forum: Python (https://sickgaming.net/forum-83.html)
+--- Thread: [Tut] 5 Best Ways to Check a List for Duplicates in Python (/thread-99446.html)

[Tut] 5 Best Ways to Check a List for Duplicates in Python - xSicKxBot - 05-23-2022

5 Best Ways to Check a List for Duplicates in Python

Rate this post

Problem Formulation and Solution Overview

In this article, you’ll learn how to check a List for Duplicates in Python.

To make it more fun, we have the following running scenario:

The Finxter Academy has given you an extensive list of usernames. Somewhere along the line, duplicate entries were added. They need you to check if their [i]List contains duplicates. For testing purposes, a small sampling of this List is used.[/i]

Question: How would we write Python code to check a List for duplicate elements?

We can accomplish this task by one of the following options:

Method 1: Use
Code:
set()
and
Code:
List
to return a Duplicate-Free List
Method 2: Use
Code:
set()
,
Code:
For
loop and
Code:
List
to return a List of Duplicates found.
Method 3: Use a
Code:
For
loop to return Duplicates and Counts
Method 4: Use
Code:
any()
to check for Duplicates and return a Boolean
Method 5: Use List Comprehension to return a List of all Duplicates

Method 1: Use set() and List to return a Duplicate-Free List

This method uses

Code:
set()

which removes any duplicate values (

Code:
set(users)

) to produce a Duplicate-Free

Code:
set()

. This set is then converted to a List (

Code:
list(set(users))

Code:
users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] dup_free = list(set(users))
print(dup_free)

This code declares a small sampling of Finxter usernames and saves them to

Code:
users

.

Next,

Code:
set()

is called and

Code:
users

is passed as an argument to the same. Then, the new set is converted to a

Code:
List

and saved to

Code:
dup_free

.

If

Code:
dup_free

was output to the terminal before converting to a

Code:
List

, the result would be a

Code:
set()

, which is not subscriptable. Meaning the elements are inaccessible in this format.

Output

Code:
{'csealker', 'cdriver', 'shoeguy', 'ollie3', 'kyliek', 'stewieboy', 'AmyP'}

Note: Any attempt to access an element from a set will result in a not subscriptable error.

In this example, the

Code:
set()

was converted to a

Code:
List

, and displays a

Code:
List

of Duplicate-Free values.

Output

Code:
['csealker', 'cdriver', 'shoeguy', 'ollie3', 'kyliek', 'stewieboy', 'AmyP']

Note: An empty set will result if no argument is passed.

Method 2: Use set(), For loop, and List to return a List of Duplicates Found

This method uses

Code:
set()

, and a

Code:
For

loop to check for and return any Duplicates found

Code:
(set(x for x in users if ((x in tmp) or tmp.add(x)))

) to

Code:
dups

. The

Code:
set()

is then converted to a

Code:
List

(

Code:
print(list(dups))

).

Here’s an example:

Code:
users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] tmp = set()
dups = set(x for x in users if (x in tmp or tmp.add(x)))
print(list(dups))

This code declares a small sampling of Finxter usernames and saves them to

Code:
users

.

Next, a new empty set,

Code:
tmp

is declared. A

Code:
For

loop is then instantiated to check each element in

Code:
users

for duplicates. If a duplicate is found, it is appended to

Code:
tmp

. The results save to

Code:
dups

as a

Code:
set()

.

Output

In this example, the

Code:
set()

was converted to a

Code:
List

and displays a

Code:
List

of Duplicates values found in the original

Code:
List

Code:
users

Code:
['kyliek', 'ollie3', 'shoeguy']

Method 3: Use a

Code:
For

loop to return [b]Duplicates and Counts[/b]

This method uses a

Code:
For

loop to navigate through and check each element of

Code:
users

while keeping track of all usernames and the number of times they appear. A Dictionary of Duplicates, including the Usernames and Counts returns.

Here’s an example:

Code:
count = {}
dup_count = {}
for i in users: if i not in count: count[i] = 1 else: count[i] += 1 dup_count[i] = count[i]
print(dup_count)

This code declares two (2) empty sets,

Code:
count

and

Code:
dup_count

respectively.

A

Code:
For

loop is instantiated to loop through each element of

Code:
users

and does the following:

If the element
Code:
i
is not in
Code:
count
, then the
Code:
count
element (
Code:
count[i]=1
) is set to one (1).
If element
Code:
i
is found in
Code:
count
, it falls to
Code:
else
where one (1) is added to
Code:
count
(
Code:
count[i]+=1
) and then added to
Code:
dup_count
(
Code:
dup_count[i]=count[i]
)

This code repeats until the end of

Code:
users

has been reached.

At this point, a Dictionary containing the Duplicates, and the number of times they appear displays.

Output

Code:
{'ollie3': 2, 'shoeguy': 2, 'kyliek': 2}

Method 4: Use Any to Check for Duplicate Values

This example uses

Code:
any()

, and passes the iterable

Code:
users

to iterate and locate Duplicates. If found,

Code:
True

returns. Otherwise,

Code:
False

returns. Best used on small Lists.

Code:
users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] dups = any(users.count(x) > 1 for x in users)
print(dups)

This code declares a small sampling of Finxter usernames and saves them to

Code:
users

.

Next,

Code:
any()

is called and loops through each element of

Code:
users

checking to see if the element is a duplicate. If found,

Code:
True

is assigned. Otherwise,

Code:
False

is assigned. The result saves to

Code:
dups

and the output displays as follows:

Output

True

Method 5: Use List Comprehension to return a [b]List of all Duplicates[/b]

This method uses List Comprehension to loop through

Code:
users

, checking for duplicates. If found, the Duplicates are appended to

Code:
dups

.

Here’s an example:

Code:
users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] dups = [x for x in users if users.count(x) >= 2]
print(dups)

This code declares a small sampling of Finxter usernames and saves them to

Code:
users

.

Next, List Comprehension extracts and displays duplicate usernames and save them to a List. The duplicate values are output to the terminal

Output

Code:
['ollie3', 'shoeguy', 'kyliek', 'ollie3', 'shoeguy', 'kyliek']

Summary

These five (5) methods of checking a List for Duplicates should give you enough information to select the best one for your coding requirements.

Good Luck & Happy Coding!