Out of HeapSpace with Matrix library

Question

I'm trying to apply a Kalman Filter to sensor readings using Java, but the matrix manipulation library I'm using is giving me a heapspace error. So, does anyone know of a matrix manipulation library for the JVM with better memory allocation characteristics?

It would seem that this one -- http://code.google.com/p/efficient-java-matrix-library/ -- is "efficient" only in name. The data set has 9424 rows by 2 columns, all values are doubles (timestamp and one dimension out of 3 on readings from a sensor).

Many thanks, guys!

Answer 1

1) The Kalman filter should not require massive, non linear scaling amounts of memory : it is only calculating the estimates based on 2 values - the initial value, and the previous value. Thus, you should expect that the amount of memory you will need should be proportional to the total amount of data points. See : http://rsbweb.nih.gov/ij/plugins/kalman.html

2) Switching over to floats will 1/2 the memory required for your calculation . That will probably be insignificant in your case - I assume that if the data set is crashing due to memory, you are running your JVM with a very small amount of memory or you have a massive data set.

3) If you really have a large data set ( > 1G ) and halving it is important, the library you mentioned can be refactored to only use floats.

4) For a comparison of java matrix libraries, you can checkout http://code.google.com/p/java-matrix-benchmark/wiki/MemoryResults_2012_02 --- the lowest memory footprint libs are ojAlgo, EJML, and Colt. Ive had excellent luck with Colt for large scale calculations - but I'm not sure which ones implement the Kalaman method.