Using converter function in genfromtxt fails

Question

When I try to read a space separated file using genfromtxt and using a converter function to convert numbers with a comma as decimal separator, I get a type error. It seems there is something wrong with my converter function. However, when I use it on a single value, it does work properly.

This is my code (I'm using Matplotlib/Pylab):

t = dtype([('Date', 'U12'), ('Time', 'U10'), ('Cond', 'f4'), ('Temp', 'f4')])

conv = lambda valstr: float(valstr.replace(',','.'))

c = {2:conv, 3:conv}

data = genfromtxt('Example.csv', dtype = t,
    skip_header=1, delimiter = ' ', converters = c)

The data looks like this:

Date Time Cond Temp
11-10-2012 00:00:14 5,430583 29,5107
11-10-2012 00:00:15 5,431812 29,45066
11-10-2012 00:00:16 5,435501 29,43862
11-10-2012 00:00:17 5,436732 29,43862
...

And this is part of the error message:

---

TypeError                                 Traceback (most recent call last)
<ipython-input-41-c65c2d17c55d> in <module>()
      5 c = {2:conv, 3:conv}
      6 
----> 7 data = genfromtxt('Example.csv', dtype = t, skip_header=1, delimiter = ' ', converters = c)


...


<ipython-input-41-c65c2d17c55d> in <lambda>(valstr)
      1 t = dtype([('Date', 'U12'), ('Time', 'U10'), ('Cond', 'f4'), ('Temp', 'f4')])
      2 
----> 3 conv = lambda valstr: float(valstr.replace(',','.'))
      4 
      5 c = {2:conv, 3:conv}

TypeError: expected an object with the buffer interface

Am I doing something wrong here, or is this some kind of bug in genfromtxt?

I'am using Python 3.2 on Win7 x64. Numpy version is 1.6.2.

Answer 1

Apparently, genfromtxt feeds read columns to converter functions as byte strings, not as unicode strings.

The problem was solved for me by changing the code of the converter function as follows:

conv = lambda valstr: float(valstr.decode("utf-8").replace(',','.'))