Store datetimes in HDF5 with H5Py

Question

How can I store NumPy datetime objects in HDF5 using h5py?

In [1]: import h5py

In [2]: import numpy as np

In [3]: f = h5py.File('foo.hdfs', 'w')

In [4]: d = f.create_dataset('data', shape=(2, 2), dtype=np.datetime64)
TypeError: No conversion path for dtype: dtype('<M8')

Answer 1

Currently HDF5 does not provide a time type (H5T_TIME is now unsupported), so there is no obvious mapping for datetime64.

One of the design goals for h5py was to stick within the base HDF5 feature set. This allows people to write data to their files and know that it will round-trip and be retrievable by people using other HDF5-aware applications like IDL and Matlab. We have made some minor exceptions before; for example, NumPy bools and complex numbers are mapped to HDF5 enums and compound types, respectively. But datetime64 seems like a much more complicated animal.

Unless there is a compelling proposal which ensures that (1) the information round-trips and (2) other HDF5 clients can reasonably make sense of it, I think we're not going to implement native support for datetime64.

In HDF5, people generally store their dates/times as string values using some variant of the ISO date format. You might consider that as a workaround.

See also: https://github.com/h5py/h5py/issues/443

Answer 2

Currently h5py does not support a time type (FAQ, Issue).

NumPy datetime64s are 8 bytes long. So as a workaround, you could view the data as '<i8' store the ints in the hdf5 file, and view it as a np.datetime64 upon retrieval:

import numpy as np
import h5py

arr = np.linspace(0, 10000, 4).astype('<i8').view('<M8[D]').reshape((2,2))
print(arr)
# [['1970-01-01' '1979-02-16']
#  ['1988-04-02' '1997-05-19']]
with h5py.File('/tmp/out.h5', "w") as f:
    dset = f.create_dataset('data', (2, 2), '<i8')
    dset[:,:] = arr.view('<i8')
with h5py.File('/tmp/out.h5', "r") as f:
    dset = f.get('data')
    print(dset.value.view('<M8[D]'))
    # [['1970-01-01' '1979-02-16']
    #  ['1988-04-02' '1997-05-19']]