You're trying to allocate 100000*100000
elements of 8 bytes each, or 80,000,000,000 bytes (74.5GB), which is failing as you only have 16GB of memory. This causes the memory allocation to fail, as it can't find a single continuous block of memory that large.
There is no fixed limit in Eigen, but the array does need to be allocatable on your system.