armadillo requested size is too large

Question

I am using armadillo4.300.0. I am operating on a dense matrix of size 2840260x103. I am loading this matrix from a .csv file of size approximately 3.7GB. I have enabled "ARMA_64BIT_WORD" in my application as well as config.hpp under armadillo_bits directory.

#if !defined(ARMA_64BIT_WORD)  
  #define ARMA_64BIT_WORD  
#endif   

I am compiling with gcc49 and running on ubuntu 12.04. When I run I am getting the following error. Interestingly, the application occasionally runs too. For eg., if I keep trying for some 10 times, it runs sometime.

error: Mat::init(): requested size is too large  

terminate called after throwing an instance of 'std::logic_error'  
  what():  Mat::init(): requested size is too large

Do I need to take care of something else?

Ramki.

Answer 1

This problem is solved with the Intel MKL library, when we compile with the -DMKL_ILP64 -m64. Typically we focus only on link flags. But it is important to note that these flags must be enabled during compile phase on the gcc command as well. I am not sure how to enable this on openmpi library. Also the lib armadillo.so must link with mkl_ilp64 instead of mkl_lp64. Follow the instruction below.

Building and installing armadillo :

• export CXX=icpc
• export CC=icpc
• export PATH=$PATH:/home/ramki/intel/bin:
• Edit $armadillo_root/cmake_aux/Modules/ARMA_FindMKL.cmake, include the PATHS correctly.
• Edit $armadillo_root/cmake_aux/Modules/ARMA_FindMKL.cmake, change mkl_lp64 to mkl_ilp64
• Edit $armadillo_root/CMakeLists.txt and (1) Change CMAKE_SHARED_LINKER_FLAGS to include the link line by intel link advisor and (2) Change CMAKE_CXX_FLAGS as given by intel link advisor
• Run ./configure and make sure MKL library is used for blas and lapack, icpc to be the compiler and the rest to be alright.
• Run make .
• Verify the linked libraries by running ldd libarmadillo.so. Mainly verify whether it is linked with mkl_ilp64 library and mkl blas and lapack libraries.
• Now run make install DESTDIR=local path.

This should work.