numpy how to load list of tuples having integers

Question

I want to load buffer string which is of type list of tuples into numpy array.

e.g.

numpy.fromstring('(1,2),(3,4),', dtype=numpy.int64)

ValueError: string size must be a multiple of element size

However i'm not able to determine which dtype argument shall I use ?

I don't want to go for numpy.array(eval('(1,2),(3,4),')) as length of tuple would be bit higher in some cases...

Answer 1

Perhaps you could use re.finditer with np.fromiter:

import re
import numpy as np

text = '(1,2),(3,4),'
arr = np.fromiter((item.group() 
                   for item in 
                   re.finditer(r'\d+', text)),
                  dtype=np.int64).reshape((-1, 2))

print(arr)

yields

array([[1, 2],
       [3, 4]], dtype=int64)

---

Note that if you could arrange for the text to be:

'1 2\n3 4'

Then it could be loaded into a NumPy array much more easily (and faster):

import io
text = '1 2\n3 4'
np.genfromtxt(io.BytesIO(text), dtype=np.int64)

yields

array([[1, 2],
       [3, 4]], dtype=int64)

Answer 2

Will this do the trick for you?

import ast
import numpy as np

mystring = '(1,2),(3,4),(5,6), (7,8,9), (10,11,12,13)'
numpy.array(ast.literal_eval(mystring))

Result

array([(1, 2),
       (3, 4),
       (5, 6),
       (7, 8, 9),
       (10, 11, 12, 13)],
     dtype=object)

Answer 3

The error is happening because numpy is expecting your string to represent the bytes which make up the data. In order to have numpy interpret your data as text, you'd need to provide the sep argument.

However, that still won't help you. You have multiple characters here which are your separators. I think that the best you can do is ast.literal_eval the string into a tuple of tuples and pass that to numpy.array. The benefit of this is that you get a 2D array off the bat -- I don't believe you can coax numpy.fromstring into giving you a 2D array -- You'd have to somehow figure out the dimensions and reshape it.

Ultimately, if the eval or ast.literal_eval step is a bottleneck, then you probably should start to think about how you could restructure the code/input