Extracting specific values from .npy file

Question

I have a .npy file of which I know basically everything (size, number of elements, type of elements, etc.) and I'd like to have a way to retrieve specific values without loading the array. The goal is to use the less amount of memory possible.

I'm looking for something like

def extract('test.npy',i,j):
    return "test.npy[i,j]"

I kinda know how to do it with a text file (see recent questions) but doing this with a npy array would allow me to do more than line extraction.

Also if you know any way to do this with a scipy sparse matrix that would be really great.

Thank you.

Answer 1

Just use data = np.load(filename, mmap_mode='r') (or one of the other modes, if you need to change specific elements, as well).

This will return a memory-mapped array. The contents of the array won't be loaded into memory and will be on disk, but you can access individual items by indexing the array as you normally would. (Be aware that accessing some slices will take much longer than accessing other slices depending on the shape and order of your array.)

HDF is a more efficient format for this, but the .npy format is designed to allow for memmapped arrays.