forrtl: severe (179): Cannot allocate array - overflow on array size calculation

Question

I am trying to run a simulation program on my laptop (Linux 3.8.0-25-generic x86_64, with Ubuntu 13.04).

It compiles okay, but when getting to allocation of some array sizes I get:

forrtl: severe (179): Cannot allocate array - overflow on array size calculation.\

Some googling about this message led me to conclude that this is generated because my program is out of memory.

To check this assumption I tried to allocate the same array a smaller dimension, but still got the same problem.

What I tried next was to increase my stack memory, but still got the same problem.

This is the code:

program memoria

implicit none

integer :: n,num
complex, dimension(:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:), allocatable :: ddptrj

n=4

num=21


allocate(ddptrj(n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,num))

deallocate(ddptrj)


endprogram memoria

How can I solve this problem?

Answer 1

The Fortran 2008 Standard defines a maximum rank of 15. ifort allows 31. gfortran still does not support more than 7 dimensions as of May 2013 . So - if you really need all 22 dimensions, use ifort - but be aware that this feature is an extension of the standard, other compilers might not compile your code.

EDIT: To sum up the discussion in the comments:

There are two problems - one related to the maximum dimensions and one related to the array size.

For compilers that do not allow for 22 dimensions, one could use one large array of rank 1 that holds all elements and calculate the index inside the element oneself using strides.

Internally all arrays are stored contiguous in memory - think of it as a large array of rank one and size N*M*... Fortran actually stores the resulting array in column-major format. So a matrix Z(NxM) is stored as a vector A(N*M), and access to the elements is calculated by strides: Z(3,4) is at index A( M*(4-1)+3 ). Note that the memory is the same for both arrays!

The next problem is the array size:

For n=2 the code works because than it fits into memory: 2**21*21=44040192 at complex (16 bytes) this amounts to 672 MB. At n=4 this becomes 4**21*21 which is 1344TB.

Answer 2

If you're using a method that requires 22 dimensions then you almost certainly need to change your algorithm. Not only will you use enormous amounts of memory, as people have noted elsewhere, but the runtime itself will become absolutely enormous if you want to do something with all the elements in that array.

Because you're using Fortran, my guess is that you are looking at some kind of parameter space with 22 dimensions in, and recording a goodness of fit or likelihood for each combination of parameters. In that case you should consider other algorithms like MCMC for exploring the space.

But this is just a guess - if you tell us what underlying problem you're trying to solve people may have more suggestions.