Question

I have a 1043*261 matrix with very small numbers between 0 and 1, and I calculated 1043*1043 covariance matrix using numpy.cov() function. I tried to run the code a few times and got similar (not exactly the same) covariance matrices, but the elements in the covariance matrices were slightly different by scale of e-7. This sometimes made the covariance matrix non-PSD, which will cause serious problem for me.

Does anyone know why the differences would exist and how to solve it?

Attached are two covariance matrices I got by running the same code twice. If you compare them by element, you will see slight differences:

No. 1
[[  5.05639177e-06   2.44041401e-06   3.30187175e-06 ...,   1.66634014e-06
4.03972183e-06   1.18433575e-06]
[  2.44041401e-06   9.67277658e-06   9.04356309e-06 ...,   2.50668884e-06
5.43371939e-06   4.74297546e-06]
[  3.30187175e-06   9.04356309e-06   2.09334309e-05 ...,   3.13977728e-06
8.69946165e-06   6.15981652e-06]
..., 
[  1.66634014e-06   2.50668884e-06   3.13977728e-06 ...,   4.20175297e-06
4.16076781e-06   1.59827406e-06]
[  4.03972183e-06   5.43371939e-06   8.69946165e-06 ...,   4.16076781e-06
2.58010941e-05   3.02797946e-06]
[  1.18433575e-06   4.74297546e-06   6.15981652e-06 ...,   1.59827406e-06
3.02797946e-06   6.60805238e-06]]

No.2
[[  5.05997030e-06   2.42187179e-06   3.30788097e-06 ...,   1.66495376e-06
4.03676937e-06   1.17413702e-06]
[  2.42187179e-06   9.60677140e-06   9.05219266e-06 ...,   2.50338648e-06
5.42679569e-06   4.75547515e-06]
[  3.30788097e-06   9.05219266e-06   2.04172017e-05 ...,   3.13058624e-06
8.67976701e-06   6.28137859e-06]
..., 
[  1.66495376e-06   2.50338648e-06   3.13058624e-06 ...,   4.20175297e-06
4.16076781e-06   1.59827884e-06]
[  4.03676937e-06   5.42679569e-06   8.67976701e-06 ...,   4.16076781e-06
2.58010941e-05   3.02810307e-06]
[  1.17413702e-06   4.75547515e-06   6.28137859e-06 ...,   1.59827884e-06
3.02810307e-06   6.63834973e-06]]

Thank you very much!

Was it helpful?

Solution

numpy.cov seems to be deterministic:

import numpy

randoms = numpy.random.random((1043, 261))

covs = [numpy.cov(randoms) for _ in range(10)]
all((c==covs[0]).all() for c in covs)
#>>> True

I'd imagine the problem is elsewhere.

Also note that this result holds with numbers 1000th the size

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top