Duplicate Removal in a list of list

Question

[
[0.074, 0.073, 0.072, 0.03, 0.029, 0.024, 0.021, 0.02], 
[0.02, 0.02, 0.015], 
[0.026, 0.026, 0.02, 0.02, 0.02, 0.015], 
[0.021, 0.021, 0.02, 0.017], [0.077, 0.076, 0.074, 0.055, 0.045, 0.021], 
[0.053, 0.052, 0.051, 0.023, 0.022], 
[0.016, 0.016]
]

The above is a output from a list of list, data['stock'].

I am thinking of removing the duplicate content within each sub-list but can't figure out a way to do it. If you take a look at line 3, you will notice that there are three elements (0.02, 0.02 and 0.015). However, the first 2 elements are actually duplicate and so one of the element is redundant.

Is there a way I could do a check in each sub-list to get rid of the duplicate value while preserving the order?

Please advise!

Answer 1

Looks like the sublists are already sorted, so you can apply itertools.groupby:

In [1]: data = [
   ...: [0.074, 0.073, 0.072, 0.03, 0.029, 0.024, 0.021, 0.02], 
   ...: [0.02, 0.02, 0.015], 
   ...: [0.026, 0.026, 0.02, 0.02, 0.02, 0.015], 
   ...: [0.021, 0.021, 0.02, 0.017], [0.077, 0.076, 0.074, 0.055, 0.045, 0.021], 
   ...: [0.053, 0.052, 0.051, 0.023, 0.022], 
   ...: [0.016, 0.016]
   ...: ]

In [2]: from itertools import groupby

In [3]: [[k for k, g in groupby(subl)] for subl in data]
Out[3]: 
[[0.074, 0.073, 0.072, 0.03, 0.029, 0.024, 0.021, 0.02],
 [0.02, 0.015],
 [0.026, 0.02, 0.015],
 [0.021, 0.02, 0.017],
 [0.077, 0.076, 0.074, 0.055, 0.045, 0.021],
 [0.053, 0.052, 0.051, 0.023, 0.022],
 [0.016]]

Answer 2

A solution using OrderedDict. This will work for both consecutive and non consecutive duplicates while preserving the order

>>> from collections import OrderedDict
>>> some_list = [
[0.074, 0.073, 0.072, 0.03, 0.029, 0.024, 0.021, 0.02],
[0.02, 0.02, 0.015],
[0.026, 0.026, 0.02, 0.02, 0.02, 0.015],
[0.021, 0.021, 0.02, 0.017], [0.077, 0.076, 0.074, 0.055, 0.045, 0.021],
[0.053, 0.052, 0.051, 0.023, 0.022],
[0.016, 0.016]
]
>>> [OrderedDict.fromkeys(e).keys() for e in some_list]
[[0.074, 0.073, 0.072, 0.03, 0.029, 0.024, 0.021, 0.02], [0.02, 0.015], [0.026, 0.02, 0.015], [0.021, 0.02, 0.017], [0.077, 0.076, 0.074, 0.055, 0.045, 0.021], [0.053, 0.052, 0.051, 0.023, 0.022], [0.016]]

Answer 3

Simple anwer:

use list(set(your_list)).

Set creates the unique elements and list() will convert it to list.

>>> lis1 = [1,2,3,4,4,3,2]
>>> print list(set(lis1))
[1, 2, 3, 4]
>>>