Best dtype for creating large arrays with numpy

Question

I am looking to store pixel values from satellite imagery into an array. I've been using

np.empty((image_width, image_length)

and it worked for smaller subsets of an image, but when using it on the entire image (3858 x 3743) the code terminates very quickly and all I get is an array of zeros.

I load the image values into the array using a loop and opening the image with gdal

img = gdal.Open(os.path.join(fn + "\{0}".format(fname))).ReadAsArray()

but when I include print img_array I end up with just zeros.

I have tried almost every single dtype that I could find in the numpy documentation but keep getting the same result.

Is numpy unable to load this many values or is there a way to optimize the array?

I am working with 8-bit tiff images that contain NDVI (decimal) values.

Thanks

Answer 1

Not certain what type of images you are trying to read, but in the case of radarsat-2 images you can the following:

dataset = gdal.Open("RADARSAT_2_CALIB:SIGMA0:" + inpath + "product.xml")
S_HH = dataset.GetRasterBand(1).ReadAsArray()
S_VV = dataset.GetRasterBand(2).ReadAsArray()
# gets the intensity  (Intensity = re**2+imag**2), and amplitude = sqrt(Intensity)
self.image_HH_I = numpy.real(S_HH)**2+numpy.imag(S_HH)**2
self.image_VV_I = numpy.real(S_VV)**2+numpy.imag(S_VV)**2

But that is specifically for that type of images (in this case each image contains several bands, so i need to read in each band separately with GetRasterBand(i), and than do ReadAsArray() If there is a specific GDAL driver for the type of images you want to read in, life gets very easy

If you give some more info on the type of images you want to read in, i can maybe help more specifically

Edit: did you try something like this ? (not sure if that will work on tiff, or how many bits the header is, hence the something:)

A=open(filename,"r")
B=numpy.fromfile(A,dtype='uint8')[something:].reshape(3858,3743)
C=B*1.0
A.close()

Edit: The problem is solved when using 64bit python instead of 32bit, due to memory errors at 2Gb when using the 32bit python version.