Numpy cumulative sum for csv rows gives type error

Question

I'm using cumsum function to calculate cumulative sum of a csv column(rows). Input looks like this:

input (average_):

1   0.000053    0.004531    0.010447    0.051962
2   0.000589    0.004518    0.009801    0.052226
3   0.000083    0.004581    0.010362    0.052288
4   -0.000192   0.003726    0.011258    0.051094
5   0.000281    0.004078    0.01008     0.052156
6   0.000091    0.004028    0.009853    0.052301
7   0.000478    0.004298    0.009966    0.054803
8   0.00028     0.004267    0.010051    0.050296
9   0.000198    0.004766    0.010245    0.051754

but this code:

acc_column = []
for row in average_:
    f_column = numpy.cumsum(row[1], dtype = float)
    acc_column.append(f_column)

doesn't work and gives the error:

"TypeError: cannot perform accumulate with flexible type"

Any ideas what am I doing wrong please?

Answer 1

Looks like some of your data is not the right dtype, try this (converting it to float), does it work or gives another TypeError?

f_column = numpy.cumsum(numpy.asarray(row[1], dtype=float))

Answer 2

I'd bet that what you have (even though you haven't shown us) is a structured array, something like this:

array([(1, 5.3e-05, 0.004531, 0.010447, 0.051962),
       (2, 0.000589, 0.004518, 0.009801, 0.052226),
       (3, 8.3e-05, 0.004581, 0.010362, 0.052288),
       (4, -0.000192, 0.003726, 0.011258, 0.051094),
       (5, 0.000281, 0.004078, 0.01008, 0.052156),
       (6, 9.1e-05, 0.004028, 0.009853, 0.052301),
       (7, 0.000478, 0.004298, 0.009966, 0.054803),
       (8, 0.00028, 0.004267, 0.010051, 0.050296),
       (9, 0.000198, 0.004766, 0.010245, 0.051754)], 
      dtype=[('f0', '<i8'), ('f1', '<f8'), ('f2', '<f8'), ('f3', '<f8'), ('f4', '<f8')])

If not, then you can load it from a file with:

a = np.genfromtxt('tmp.txt', dtype=None)

Once it's all in one array, there's no need to loop through each row:
If you want to accumulate column 1, then that's actually field 'f1' in a structured array, so use:

>>> np.cumsum(a['f1'])
array([ 5.300e-05, 6.420e-04, 7.250e-04,
        5.330e-04, 8.140e-04, 9.050e-04,
        1.383e-03, 1.663e-03, 1.861e-03])

If it's an ordinary float array:

array([[  1.000e+00,   5.300e-05,   4.531e-03,   1.0447e-02,   5.1962e-02],
       [  2.000e+00,   5.890e-04,   4.518e-03,   9.8010e-03,   5.2226e-02],
       [  3.000e+00,   8.300e-05,   4.581e-03,   1.0362e-02,   5.2288e-02],
       [  4.000e+00,  -1.920e-04,   3.726e-03,   1.1258e-02,   5.1094e-02],
       [  5.000e+00,   2.810e-04,   4.078e-03,   1.0080e-02,   5.2156e-02],
       [  6.000e+00,   9.100e-05,   4.028e-03,   9.8530e-03,   5.2301e-02],
       [  7.000e+00,   4.780e-04,   4.298e-03,   9.9660e-03,   5.4803e-02],
       [  8.000e+00,   2.800e-04,   4.267e-03,   1.0051e-02,   5.0296e-02],
       [  9.000e+00,   1.980e-04,   4.766e-03,   1.0245e-02,   5.1754e-02]])

Then all you have to do is call cumsum on the column:

>>> np.cumsum(a[:,1])
array([ 5.300e-05,  6.420e-04,  7.250e-04,
        5.330e-04,  8.140e-04,  9.050e-04,
        1.383e-03,  1.663e-03,  1.861e-03])