Python Empirical distribution function (ecdf) implementation

Question

I am aware of statsmodels.tools.tools.ECDF but since the calculation of an empricial cumulative distribution function (ECDF) is pretty straight-forward and I want to minimise dependencies in my project, I want to code it manually.

In a given list() / np.array() Pandas.Series, the ECDF for each element can be calculated as given in Wikipedia:

I have the Pandas DataFrame ,dfser, below and I want to get the ecdf of the values column. My two one-liner solutions are given as well.

Is there a faster way to do this? Speed is important in my application.

# Note that in my case indices are unique identifiers so I cannot reset them.
import numpy as np
import pandas as pd

# all indices are unique, but there may be duplicate measurement values (that belong to different indices). 
dfser = pd.DataFrame({'group':['a','b','b','a','d','c','e','e','c','a','b','d','d','c','d','e','e','a'],
                      'values':[2.01899E-06, 1.12186E-07, 8.97467E-07, 2.91257E-06, 1.93733E-05, 
                                0.00017889, 0.000120963, 4.27643E-07, 3.33614E-07, 2.08352E-12,  
                                1.39478E-05, 4.28255E-08, 9.7619E-06, 8.51787E-09, 1.28344E-09, 
                                3.5063E-05, 0.01732035,2.08352E-12]},
                       index = [123, 532, 235, 645, 747, 856, 345, 245, 845, 248, 901, 712, 162, 126, 
                              198,748, 127,395]      )

# My 1st Solution - list comprehension
dfser['ecdf']=[sum( dfser['values'] <= x)/float(dfser['values'].size) for x in dfser['values']]

# My 2nd Solution - ranking
dfser['rank'] = dfser['values'].rank(ascending = 0)
dfser['ecdf_r']=(len(dfser)-dfser['rank']+1)/len(dfser)
dfser
    group        values      ecdf  rank    ecdf_r
123     a  2.018990e-06  0.555556   9.0  0.555556
532     b  1.121860e-07  0.333333  13.0  0.333333
235     b  8.974670e-07  0.500000  10.0  0.500000
645     a  2.912570e-06  0.611111   8.0  0.611111
747     d  1.937330e-05  0.777778   5.0  0.777778
856     c  1.788900e-04  0.944444   2.0  0.944444
345     e  1.209630e-04  0.888889   3.0  0.888889
245     e  4.276430e-07  0.444444  11.0  0.444444
845     c  3.336140e-07  0.388889  12.0  0.388889
248     a  2.083520e-12  0.111111  17.5  0.083333
901     b  1.394780e-05  0.722222   6.0  0.722222
712     d  4.282550e-08  0.277778  14.0  0.277778
162     d  9.761900e-06  0.666667   7.0  0.666667
126     c  8.517870e-09  0.222222  15.0  0.222222
198     d  1.283440e-09  0.166667  16.0  0.166667
748     e  3.506300e-05  0.833333   4.0  0.833333
127     e  1.732035e-02  1.000000   1.0  1.000000
395     a  2.083520e-12  0.111111  17.5  0.083333

Answer 1

Since you are already using pandas I think it will be silly not to use some of its features:

In [15]:
import numpy as np
from numpy import *
sq=ser.value_counts()
sq.sort_index().cumsum()*1./len(sq)
Out[15]:
2.083520e-12    0.058824
1.283440e-09    0.117647
8.517870e-09    0.176471
4.282550e-08    0.235294
1.121860e-07    0.294118
3.336140e-07    0.352941
4.276430e-07    0.411765
8.974670e-07    0.470588
2.018990e-06    0.529412
2.912570e-06    0.588235
9.761900e-06    0.647059
1.394780e-05    0.705882
1.937330e-05    0.764706
3.506300e-05    0.823529
1.209630e-04    0.882353
1.788900e-04    0.941176
1.732035e-02    1.000000
dtype: float64

And speed comparison

In [19]:

%timeit sq.sort_index().cumsum()*1./len(sq)
1000 loops, best of 3: 344 µs per loop
In [18]:

%timeit ser.value_counts().sort_index().cumsum()*1./len(ser.value_counts())
1000 loops, best of 3: 1.58 ms per loop
In [17]:

%timeit [sum( ser <= x)/float(len(ser)) for x in ser]
100 loops, best of 3: 3.31 ms per loop

If the values are all unique, the ser.value_counts() is no longer needed. That part is slow (Fetching unique values). All you need in that case is just to sort ser.

In [23]:

%timeit np.arange(1, ser.size+1)/float(ser.size)
10000 loops, best of 3: 11.6 µs per loop

---

The fastest version that I can think of is to use get vectorized:

In [35]:

np.sum(dfser['values'].values[...,newaxis]<=dfser['values'].values.reshape((1,-1)), axis=0)*1./dfser['values'].size
Out[35]:
array([ 0.55555556,  0.33333333,  0.5       ,  0.61111111,  0.77777778,
        0.94444444,  0.88888889,  0.44444444,  0.38888889,  0.11111111,
        0.72222222,  0.27777778,  0.66666667,  0.22222222,  0.16666667,
        0.83333333,  1.        ,  0.11111111])

Add let see:

In [37]:

%timeit dfser['ecdf']=[sum( dfser['values'] <= x)/float(dfser['values'].size) for x in dfser['values']]
100 loops, best of 3: 6 ms per loop
In [38]:

%%timeit
dfser['rank'] = dfser['values'].rank(ascending = 0)
dfser['ecdf_r']=(len(dfser)-dfser['rank']+1)/len(dfser)
1000 loops, best of 3: 827 µs per loop
In [39]:

%timeit np.sum(dfser['values'].values[...,newaxis]<=dfser['values'].values.reshape((1,-1)), axis=0)*1./dfser['values'].size
10000 loops, best of 3: 91.1 µs per loop