Pytables table dtype alignment

Question

If I create the following aligned Numpy array

import numpy as np
import tables as pt
numrows = 10
dt = np.dtype([('date', [('year', '<i4'), ('month', '<i4'), ('day', '<i4')]), 
               ('apples', '<f8'), 
               ('oranges', '|S7'),
               ('pears', '<i4')], align=True) 
x = np.zeros(numrows, dtype=dt)
for d in x.dtype.descr:
    print d

and print the dtype.descr I get the following:

('date', [('year', '<i4'), ('month', '<i4'), ('day', '<i4')])
('', '|V4')
('apples', '<f8')
('oranges', '|S7')
('', '|V1')
('pears', '<i4')

The dtype includes these extra void spaces '|V4', '|V1'

Now, when I create a Pytable->table using this same dtype (Numpy flavor) it seems I lose the alignment.

h5file = pt.open_file('mytable.h5', mode='w')
table = h5file.create_table('/', 'mytable', dt, filters=None, expectedrows=numrows, byteorder='little')
policy = table.row

for j in xrange(numrows):
    for field in table.colnames:
        if (field == 'date'):
            policy[field] = (2014, 1, 8)
        else:
            policy[field] = 0
    policy.append() 

table.flush()
mytable = h5file.root.mytable[:]
h5file.close()
for d in mytable.dtype.descr:
    print d

The output of which is:

('date', [('year', '<i4'), ('month', '<i4'), ('day', '<i4')])
('apples', '<f8')
('oranges', '|S7')
('pears', '<i4')

No more '|V' spaces

How can I construct a Pytable->table such that the alignment is preserved (keeps the '|V' spaces)?

Answer 1

PyTables does not support the numpy void data type for columns by default - see the source for tables.descr_from_dtype(). However, you can probably trick PyTables into working by replacing the voids with uint8s. This would look something like:

dt = ...
expanded_dt = np.dtype(dt.descr)
newdt = []
for name, col in zip(expanded_dt.names, expanded_dt):
    if np.issubdtype(col, np.void):
        newdt.append([name, np.uint8(col.itemsize)])
    else:
        newdt.append([name, col])


newdt = np.dtype(newdt)

This would put in fake columns that have the right width.