Stacking numpy recarrays without losing their recarrayness

Question

Suppose I make two recarrays with the same dtype and stack them:

>>> import numpy as np
>>> dt = [('foo', int), ('bar', float)]
>>> a = np.empty(2, dtype=dt).view(np.recarray)
>>> b = np.empty(3, dtype=dt).view(np.recarray)
>>> c = np.hstack((a,b))

Although a and b are recarrays, c is not:

>>> c.foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'numpy.ndarray' object has no attribute 'foo'
>>> d = c.view(np.recarray)
>>> d.foo
array([                  0,     111050731618561,                   0,
                   7718048, 8246760947200437872])

I can obviously turn it into a recarray again, as shown with d above, but that is inconvenient. Is there a reason why stacking two recarrays does not produce another recarray?

Answer 1

I don't know. Most likely it's a bug/feature that's never been implemented. numpy.hstack is basically a wrapper around a function in numpy.core.fromnumeric. Numeric is one of the two predecessors of numpy. Most functions in numpy have a convention to output the same type as the input by calling the method __array_wrap__ of the input on the output, and the resulting output is should have the same data, but "wrapped" in the new class. Perhaps the concept of "wrapping" was not in numeric, andnever got added to this function.

You can use this technique to make a smarter stacking function

def hstack2(arrays) :
  return arrays[0].__array_wrap__(numpy.hstack(arrays))

This works for both recarrays and regular arrays

>>> f = hstack2((a,b))
>>> type(f)
<class 'numpy.core.records.recarray'>
>>> f.foo
array([    140633760262784,     111050731618561,     140633760262800,
                   7536928, 8391166428122670177])
>>> x = numpy.random.rand(3)
>>> y = numpy.random.rand(2)
>>> z = hstack2((x,y))
>>> type(z)
<type 'numpy.ndarray'>

I'm not sure what you're planning, but you might want to ask on the numpy mailing list is there's a better way than using the documented, but double-underscored method, and what their reasoning is for not doing the wrapping themselves.

Answer 2

Alternatively, there are some helper utilities in numpy.lib.recfunctions which I stumbled across here. This module has functions for both merging and stacking recarrays:

from numpy.lib.recfunctions import stack_arrays
c = stack_arrays((a, b), asrecarray=True, usemask=False)
c.foo
>>> array([     140239282560000,           4376479720, -4611686018427387904,
                     4358733828,           4365061216])

If one wants to add extra columns to a recarray, this can be done using merge_arrays:

import numpy as np
from numpy.lib.recfunctions import merge_arrays
dt1 = [('foo', int), ('bar', float)]
dt2 = [('foobar', int), ('barfoo', float)]
aa = np.empty(6, dtype=dt1).view(np.recarray)
bb = np.empty(6, dtype=dt2).view(np.recarray)

cc = merge_arrays((aa, bb), asrecarray=True, flatten=True)
type(cc)
>>> numpy.core.records.recarray

(Although not an answer to the question, I'm posting the latter example as a reference)

Answer 3

Incidentally, you can also use:

c = np.concatenate((a,b))

or

c = np.r_[a, b]

( Source: this mailing list message )