Python27 Memory Error Issues

Question

I'm trying to process some data utilizing python but encounter "MemoryError" frequently. For my data processing, I need to run a loop ~400 times. The issue is, every time I get 20-30 times into this loop I get "MemoryError". While watching Task Manager, this appears to happen when python.exe*32 is ~1 GB. I would use 64-bit python but some of the libraries I need are only 32 bit. Furthermore, at the end of the main function I'm calling with every loop, I'm utilizing the del function for just about every variable I use. When the "MemoryError" occurs, I have to completely exit the interpreter, otherwise if I try to re-reun it again it automatically fails with "MemoryError".

I'm using Python(x,y) 2.7 and need the following: numpy scipy.io matplotlib pyopencl pyfft

Answer 1

You said it crashed on a line like this:

s = zeros((A,B,C),complex128)

With A=2400, B=256, C=25. That would require 235 MB of memory. And not just any 235 MB: it must be contiguous, because NumPy expects to use it as a single array.

You also mentioned you're running this in a 32-bit process, and that it crashes when the process memory usage gets around 1 GB.

This is unfortunate but not entirely surprising. Consider that the total usable virtual memory in a 32-bit process is somewhere around 3 GB, and there will inevitably be some "holes", meaning you will never be able to allocate a single array 4 GB in size. But how much can you allocate in one piece? Well, that depends on the fragmentation of the memory you have allocated so far, as virtual addresses that are still needed by your program may become "sprinkled" around its 4 GB address space, and eventually you may become unable to allocate a quarter-gigabyte chunk all together.

What you should do here is to allocate your matrix once and reuse it. When your program has just started you are extremely likely to succeed in allocating the 235 MB array, and you can later clear it before reusing it without allocating again.

Alternatively you could (a) move to 64-bit, or (b) spend potentially quite a bit of time tracking down why you can't allocate a large contiguous chunk after running a garbage-collected language for a while.