loadmat python memory error

Question

I'm new to Python and I want to import a matlab struct of size 850M to it. I use "loadmat" but I get a memory error:

return self._matrix_reader.array_from_header(header, process)
  File "mio5_utils.pyx", line 624, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy\io\matlab\mio5_utils.c:5401)
  File "mio5_utils.pyx", line 653, in scipy.io.matlab.mio5_utils.VarReader5.array_from_header (scipy\io\matlab\mio5_utils.c:4849)
  File "mio5_utils.pyx", line 706, in scipy.io.matlab.mio5_utils.VarReader5.read_real_complex (scipy\io\matlab\mio5_utils.c:5578)
  File "mio5_utils.pyx", line 424, in scipy.io.matlab.mio5_utils.VarReader5.read_numeric (scipy\io\matlab\mio5_utils.c:3439)
  File "mio5_utils.pyx", line 360, in scipy.io.matlab.mio5_utils.VarReader5.read_element (scipy\io\matlab\mio5_utils.c:3164)
  File "streams.pyx", line 76, in scipy.io.matlab.streams.GenericStream.read_string (scipy\io\matlab\streams.c:1408)
MemoryError

I'm running python 3.2 on a Windows XP with 3.5G of RAM. Here is my code:

from scipy.io import matlab as mio
mat = mio.loadmat(DIR + '/input.mat')

Could you please help me and tell me what I should do to fix this?

Answer 1

You are probably using 32-bit Python. The maximum limit for all 32-bit programs (this issue in fact has nothing to do with Python or Scipy) is 2GB --- how much memory you have installed on the machine does not matter. In practice, the allocation of large objects starts to fail already earlier, due to virtual memory fragmentation (and ~800 MB seems to be expected as a rule of thumb).

The solution would be to use 64-bit Python instead of the 32-bit one. For this, you need a 64-bit operating system such as Windows 7 or Linux.

However, if you are stuck with the 32-bit version of Windows XP, there are some tricks for bumping the 32-bit memory limit up to 3GB on 32-bit systems, which may help here.