Size of the array that Fortran can handle

Question

I have 30000 files to process each file has 80000 x 5 lines. I need to read all files and process them finding the average of each line. I have written the code to read and extract all data from the file. My code is in Fortran. There is an array of (30000 X 800000) My program could not go over (3300 X 80000). I need to add the 4th column of each file in 300 file steps, I mean 4th column of 1st file with 4th column of 301st file, 4th col of 2nd file with 4th col of 302nd file and so on .Do you think this is because of the limitation of the size of array that Fortran can handle? If so, is there any way to increase the size of the array that Fortran can handle? What about the no of files? My code looks like this: This program runs well.

    implicit double precision (a-h,o-z),integer(i-n)
    dimension x(78805,5),y(78805,5),den(78805,5)
    dimension b(3300,78805),bb(78805)
    character*70,fn 
    nf = 3300       ! NUMBER OF FILES
    nj = 78804      ! Number of rows in file.
    ns = 300        ! No. of steps for files.
    ncores = 11 ! No of Cores
c--------------------------------------------------------------------
c--------------------------------------------------------------------   
    !Initialization     
    do i = 0,nf
      do j = 1, nj
        x(j,1) = 0.0
        y(j,2) = 0.0
        den(j,4) = 0.0
c       a(i,j) = 0.0
        b(i,j) = 0.0
c       aa(j)  = 0.0
        bb(j)  = 0.0
      end do
    end do
c-------!Body program-----------------------------------------------
    iout = 6    ! Output Files upto "ns" no.
    DO i= 1,nf  ! LOOP FOR THE NUMBER OF FILES
      write(fn,10)i
      open(1,file=fn)
      do j=1,nj     ! Loop for the no of rows in the domain
        read(1,*)x(j,1),y(j,2),den(j,4)
        if(i.le.ns) then
c          a(i,j) = prob(j,3)
           b(i,j) = den(j,4)
        else
c          a(i,j) = prob(j,3) + a(i-ns,j) 
           b(i,j) = den(j,4) + b(i-ns,j) 
        end if
      end do
      close(1)
c         ----------------------------------------------------------
c         -----Write Out put [Probability and density matrix]-------
c         ----------------------------------------------------------
      if(i.ge.(nf-ns)) then
        do j = 1, nj
c         aa(j) = a(i,j)/(ncores*1.0)
          bb(j) = b(i,j)/(ncores*1.0)
          write(iout,*) int(x(j,1)),int(y(j,2)),bb(j)
        end do
        close(iout)
        iout = iout + 1
      end if
    END DO
   10  format(i0,'.txt')    
       END 

Answer 1

It's hard to say for sure because you haven't given all the details yet, but your problem is quite possibly that you are using a 32 bit compiler producing 32 bit executables and you are simply running out of address space.

Although your operating system supports 64 bit address space, your 32 bit process is still limited to 32 bit addresses.

You have found a limit at 3300*78805*8 which is just under 2GB and this supports my theory.

No matter what is the cause of your immediate problem, your fundamental problem is that you appear to be loading everything into memory at once. I've not closely studied your algorithm but on first inspection it seems likely that you could re-arrange it to avoid having everything in memory at once.