Why does SciPy's loadmat throw a MemoryError when reading a 200 MB Matlab struct

Question

I'm using the following code to try to load a MAT file in Python. I can load it without issue in MATLAB.

from scipy.io import loadmat
test_filename = 'test_data.mat' #This is a struct
data =loadmat(test_filename, struct_as_record=True)

Running that code produces this error:

Traceback (most recent call last):
  File "C:\Users\mac389\workspace\nexUtils\src\qA.py", line 16, in <module>
data =loadmat(test_filename, struct_as_record=True)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio.py", line 175, in loadmat
matfile_dict = MR.get_variables(variable_names)
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 272, in get_variables
hdr, next_position = self.read_var_header()
  File "C:\Python27\lib\site-packages\scipy\io\matlab\mio5.py", line 224, in read_var_header
stream = BytesIO(dcor.decompress(data))
MemoryError

For reference, test_data.mat is a structure with the following fields (from MATLAB console):

 version: 101
 comment: 'molecular layer 4/17'
    freq: 40000
    tbeg: 0
    tend: 1.3950e+003
  events: {3x1 cell}
 neurons: {50x1 cell}
   waves: {102x1 cell}
contvars: {64x1 cell}

Test_data.mat is 217 MB. I have 4 GB of RAM. I am using SciPy 0.10.0 and NumPy 1.6.1. Changing the 'struct_as_record' field does nothing.

How can I load a struct where the fields are cell arrays?

Answer 1

I found the answer.

Loadmat can't deal with heavily nested structures. In the data set I was given, three of the struct fields, 'waves, neurons, contvars' were cell arrays. Each member of that cell array was a struct. Some of the fields of those structs were themselves cell arrays. Those cell arrays had one field that contained the data. This nonstandard way of organizing the data combined with a lack of documentation created the problem.

I guess this serves as a cautionary tale to stick as close to text file format as possible, if you are creating the data storage format, If you choose a really nonstandard format take mercy on your successor and document that fact...

Answer 2

I think it takes more memory in Python because of the way decompression is implemented. Try saving in Matlab without compression (by using -v6, the version 6 format has no compression feature).