Is it possible to mmap a recarray in python 2.7?

Question

I have a large global recarray totaling 30GBs of data in a programme running via qsub on a cluster with 256GBs of RAM. I am currently the only user on this cluster so there are no conflicts with the allocation of RAM. When looping over this recarray the system appears to shunt the object to the disc, not keep it held in RAM, thus slowing the loop in excess of 5 fold. I have been looking at using mmap on the object in the following ways and received the following errors.

m = mmap.mmap(myrecarray, 0)
MMAP TypeError: only length-1 arrays can be converted to Python scalars

m = mmap.mmap(myrecarray.fileno(), 0)
AttributeError: record array has no attribute fileno

is it possible to use mmap to hold a recarray object in the RAM, or is this a total misuse of the mmap method or can this only be done for other object types, like strings or files?

many thanks

Answer 1

This qustion has been answered here...

http://www.biostars.org/p/92704/#92764

essentially this is a total misuse of the mmap method. It only appears to be useful for files or strings, not any complex data structure.

Answer 2

Instead of mmap you may want to use numpy.memmap or numpy.load to open data file.

Or check out answer to "using numpy.memmap to map a device file" for alternative approach.