how to set dtype for nested numpy ndarray?

Question

I am working on the following data structure, from which I am trying to create a ndarray contains all the data:

      instrument         filter             response
-----------------------------------------------------
       spire              250um           array of response
         ...               ...                ...

where the array of response is:
      linenumber      wavelangth      throughput
-----------------------------------------------------
         0     1.894740e+06           0.000e+00
         1     2.000000e+06           1.000e-02
         2     2.026320e+06           3.799e-02
        ...              ....              ....

So, I hope I can turn the data to one ndarray, by using the following code:

import numpy as np

data = [('spire', '250um', [(0, 1.89e6, 0.0), (1,2e6, 1e-2), (2,2.02e6,3.8e-2), ...]),
        ('spire', '350', [ (...), (...), ...]),
        ...,
        ]
table = np.array(data, dtype=[('instrument', '|S32'),
                               ('filter', '|S64'),
                               ('response', [('linenumber', 'i'),
                                             ('wavelength', 'f'),
                                             ('throughput', 'f')])
                              ])

This code raises exception because there is list(tuple, list(tuple)) pattern. After changing the data to:

 data = [('spire', '250um', np.array([(0, 1.89e6, 0.0), (1,2e6, 1e-2), (2,2.02e6,3.8e-2), ...],
                                     dtype=[('linenumber','i'), ('wavelength','f'), ('throughput','f')])),
        ('spire', '350', np.array([ (...), (...), ...],dtype=[...])),
        ...,
        ]]

Then the code can run through, However, the result is wrong because for the response field, only the first entry of the array of response is taken:

>>print table[0]

('spire', '250um', (0,1.89e6,0.0))

instead of the whole array.

My question is, how to properly set the dtype keyword to make this work? in both cases: 1. a nested list of tuples in which list of tuples is contained; 2. a nested list of tuples in which an inhomogeneous ndarray is contained.

Thank you in advance!

Answer 1

I can get this to work if the response array is of fixed length (perhaps Numpy has to be able to precompute the size of each record in a structured array?). As noted on the Numpy manual page for structured arrays, you can specify the shape for a field in a structured array.

import numpy as np

data = [('spire', '250um', [(0, 1.89e6, 0.0), (1, 2e6, 1e-2)]),
        ('spire', '350',   [(0, 1.89e6, 0.0), (2, 2.02e6, 3.8e-2)])
        ]
table = np.array(data, dtype=[('instrument', '|S32'),
                               ('filter', '|S64'),
                               ('response', [('linenumber', 'i'),
                                             ('wavelength', 'f'),
                                             ('throughput', 'f')], (2,))
                              ])

print table[0]
# gives ('spire', '250um', [(0, 1890000.0, 0.0), (1, 2000000.0, 0.009999999776482582)])