Remove duplicate items in list, Python.

How to remove duplicate list items in python ?

Introduction

There are some options and features to remove duplicate items in a list. We are going to investigate some of them without any module (pandas, numpy etc.) in this essay.
Let's code !

Step by Step: Remove duplicate items from list

To be able to do that firstly we are going to use one of most common way, that is iteration by for loop. Assume we have a list including duplicate items like this list = [1, 2, 2, 3, 4, 5, 5, ...]. So we can use an iterator in a loop to remove duplicate items from list.

   def remove_duplicate(dup_list) -> None:
       result_list = list()
       [result_list.append(i) for i in dup_list if i not in result_list]
       print(f"Result -> {result_list}")

When the code above is run in the main method the output will be like following:

   if __name__ == "__main__":
      remove_duplicate(dup_list = [1, 2, 2, 3, 3, 4, 5])

   Result List -> [1, 2, 3, 4, 5]

As you see above code we have used the optimized way to create list. Anyways let's try the other options that we can. One of the others is removing item from the original list by count() method. Think about that like the reverse of before instance. We will not create a new list, just delete duplicate items.

   def remove_duplicate(dup_list) -> None:
       for i in dup_list[:]:
           if dup_list.count(i) > 1:
               dup_list.remove(i)
       print(f"Result List => {dup_list}")

  if __name__ == "__main__":
     remove_duplicate(dup_list = [1, 2, 2, 3, 3, 4, 5])

   Output:
   Result List -> [1, 2, 3, 4, 5]

Note that we use the list like dup_list[:] in for loop because we want to remove items without any skipping. Unless you use the whole copy of the list as we do the iterator will skip the item after remove.

So far so good ! Now we will use dictionary to remove items. To do that we need fromkeys() method however, If you don't know what that module is you can search on the net. There are two parameters for this module and those are sequence and value. Value parameter is optional so it depends on what you need.

Syntax:

dict.fromkeys(sequence, value(optional))

   result = list(dict.fromkeys(dup_list))
   print(f"Result List => {result}")

Check the output out.

   Output:
   Result List -> [1, 2, 3, 4, 5]

There is one more way to do that similar to the method above but this is slower than the fromkeys method.

   convert_to_set = list(set(dup_list))
   convert_to_set.sort(key=dup_list.index)
   print(f"Result List => {convert_to_set}")

   Output:
   Result List -> [1, 2, 3, 4, 5]

That's all in this tutorial. I think the fastest way is fromkeys() method. Compare them each other and share your comments.

Thank you for reading. :)