MemoryError from numpy.linalg.svd for large matrices

StackOverflow https://stackoverflow.com/questions/20407548

  •  29-08-2022
  •  | 
  •  

Question

The following command will fail on my machine, with Windows 7 ultimate SP1 x64, Python 3.3.3 x64, numpy 1.8.0, and 16GB memory, which seems sufficient for the task. And it also fails on a cluster.

python -c "import numpy as np;np.linalg.svd(np.random.random((25000,10000)))"

Any ideas about the reason? Thanks.

Was it helpful?

Solution

I think you are just hitting a limit of your RAM: I run this on my Linux box (64Gb of RAM) and the process occupies 14.4 Gb. If you have a 16Gb machine then some of it is occupied by OS and other stuff, so perhaps you just miss a bit of spare RAM for the process.

OTHER TIPS

My thought would be you have insufficient resources available. This ran on my Mac Mini with 16GB of RAM. Python peaked at 8.5GB and my swapfile grew to almost 15GB during processing.

What is the size of your swapfile and is it dynamic or preallocated? I would consider bumping it to 32GB, preallocated, if you have disk available.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top