https://stackoverflow.com/questions/23531704
italiano
english
français
española
中国
日本の
العربية
Deutsch
한국어
Português
RussianSince you stated you "can't deal with requiring large ammounts of RAM to use this app" your only option is to store the big array off RAM - disk being the most obvious choice (using a relational database is just an unnecessary overhead).
You can use a little utility class which provides a persistent 2-dimensional double array functionality. Here is my solution to that using RandomAccessFile. This solution also has the advantage that you can keep the array and reuse it when you restart the application!
Note: the presented solution is not thread-safe. Synchronization needed if you want to access it from multiple threads concurrently.
Persistent 2-dimensional double array:
public class FileDoubleMatrix implements Closeable {
private final int rows;
private final int cols;
private final long rowSize;
private final RandomAccessFile raf;
public FileDoubleMatrix(File f, int rows, int cols) throws IOException {
if (rows < 0 || cols < 0)
throw new IllegalArgumentException(
"Rows and cols cannot be negative!");
this.rows = rows;
this.cols = cols;
rowSize = cols * 8;
raf = new RandomAccessFile(f, "rw");
raf.setLength(rowSize * cols);
}
/**
* Absolute get method.
*/
public double get(int row, int col) throws IOException {
pos(row, col);
return get();
}
/**
* Absolute set method.
*/
public void set(int row, int col, double value) throws IOException {
pos(row, col);
set(value);
}
public void pos(int row, int col) throws IOException {
if (row < 0 || col < 0 || row >= rows || col >= cols)
throw new IllegalArgumentException("Invalid row or col!");
raf.seek(row * rowSize + col * 8);
}
/**
* Relative get method. Useful if you want to go though the whole array or
* though a continuous part, use {@link #pos(int, int)} to position.
*/
public double get() throws IOException {
return raf.readDouble();
}
/**
* Relative set method. Useful if you want to go though the whole array or
* though a continuous part, use {@link #pos(int, int)} to position.
*/
public void set(double value) throws IOException {
raf.writeDouble(value);
}
public int getRows() { return rows; }
public int getCols() { return cols; }
@Override
public void close() throws IOException {
raf.close();
}
}
The presented FileDoubleMatrix supports relative get() and set() methods which is very useful if you process your whole array or a continuous part of it (e.g. you iterate over it). Use the relative methods when you can for faster operations.
Example using the FileDoubleMatrix:
final int rows = 10;
final int cols = 10;
try (FileDoubleMatrix arr = new FileDoubleMatrix(
new File("array.dat"), rows, cols)) {
System.out.println("BEFORE:");
for (int row = 0; row < rows; row++) {
for (int col = 0; col < cols; col++) {
System.out.print(arr.get(row, col) + " ");
}
System.out.println();
}
// Process array; here we increment the values
for (int row = 0; row < rows; row++)
for (int col = 0; col < cols; col++)
arr.set(row, col, arr.get(row, col) + (row * cols + col));
System.out.println("\nAFTER:");
for (int row = 0; row < rows; row++) {
for (int col = 0; col < cols; col++)
System.out.print(arr.get(row, col) + " ");
System.out.println();
}
} catch (IOException e) {
e.printStackTrace();
}
More about the relative get and set methods:
The absolute get and set methods require the position (row and column) of the element to be returned or set. The relative get and set methods do not require the position, they return or set the current element. The current element is in fact the pointer of the underlying file. The position can be set with the pos() method.
Whenever a relative get() or set() method is called, after returning they implicitly move the pointer to the next element, in a row-continuity manner (moving to the next in the row, and if the end of row reached, moving to the first element of the next row etc.)
For example here is how we can zero the whole array using the relative set method:
// Fill the whole array with zeros using relative set
// First position to the beginning:
arr.pos(0, 0);
// And execute a "set zero" operation
// as many times as many elements the array has:
for ( int i = rows * cols; i > 0; i--)
arr.set(0);
The relative get and set methods automatically move the pointer to the next element.
It should be obvious that in my implementation the absolute get and set methods also change the pointer which must not be forgotten when relative and absolute get/set methods are used.
Another example: let's set the sum of each row to the last element of the row, but also include the last element in the sum! For this we will use the mixture of realtive and absolute get/set methods:
// Start with the first row:
arr.pos(0, 0);
for (int row = 0; row < rows; row++) {
double sum = 0;
for (int col = 0; col < cols; col++)
sum += arr.get(); // Relative get to calculate row sum
// Now set the sum to the end of row.
// For this we have to position back, so we use the absolute set.
arr.set(row, cols - 1, sum);
// The absolute set method also moves the pointer, and since
// it is the end of row, it moves to the first of the next row.
}
And that's all. Using the relative get/set methods we don't have to pass the "matrix indices" when processing continuous parts of the array, and also the implementation does not have to move the internal pointer which is more than handy when processing millions of elements as in your example.
