Memory Error when creating large lists or dictionaries or arrays

Question

I need to read a large GIS raster file (162000 * 105000). I want to convert this raster to a Numpy array (2*2 and double type) and then use that numpy array for my several analysis. My OS is windows 64 and I have 32Gb memory. However, python is not able to use the memory and I get 'Memory Error' Is there any way to allocate more memory to Python to avoid this error? Thanks

Answer 1

32Gb is not enough to hold your image so it is not possible for you to load the image entirely into your computer's memory.

With each pixel taking up 64 bits (if it is single channel, or multiply by d for d dimensional hyperspectral image) you require approximately 162000 * 105000 * 64 (bits) / (1024^3 bits) = 1013 Gbits (or roughly 126Gb) memory for your image.

You are better off using a GIS image processing library that can perform your image processing tasks out-of-core or only loading in specific subregions of the image. Geoprocessing is a very specific field within image processing and it is best that you use the appropriate libraries for the job. Check this post for more details on using the gdal library for geoprocessing.