OOM error when appending String in a for loop within a while loop

Question

For a simulation I want to be able to have a while loop that goes up to 10000 time steps. At each time point I want to store instance variables of my items. In my real code the items are objects that undergo transitions at each time point but this OOM error is caused because the String in my code is eating up my heap memory and not because of the transitions of my items.

At each time point I want to write out the instance variables of my items to a StringBuilder which I then transfer to a String array outside the for loop but within the while (time) loop.

I use a for loop to iterate through all the items and append all instance variables in one line separated by '%' to be able to delimit the data for later use.

If I use up to 5000 time steps and 1000 items my code works fine. But as soon as I increase the time steps to 6000 I get an out of memory error.

Here is my code

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;

public class Test2 {
    public static void main(String[] args) throws IOException {

        int time = 0;
        int endTime = 6000;
        int items = 1000;
        File FILE = new File("test.txt");

        String[] arrayString = new String[endTime];

        while (time < endTime) {
            StringBuilder sb = new StringBuilder(50 * items);

            for (int i = 0; i < items; i++) {

                sb.append("Data from my items I want to store in a textfile.")
                  .append("%");         // '%' functions as a split-symbol between
                                        // two items that I later use as
                                        // delimiter for loading the text file
            }

            arrayString[(int) time] = sb.toString();
            sb.delete(0, sb.length());
            sb.setLength(50 * items);

            time++;
        }
        PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter(
                FILE, true)));
        for (int j = 0; j < endTime; j++) {
            pw.println(arrayString[j]);
        }
        pw.close();
        System.out.println("Done");
    }
}

And this is the stack trace

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Unknown Source)
    at java.lang.String.<init>(Unknown Source)
    at java.lang.StringBuilder.toString(Unknown Source)
    at package2.Test2.main(Test2.java:30)

and the line of code it points at

arrayString[(int) time] = sb.toString();

Did I reach the limit of how much String you can store in a String array? Or is there some way to improve (the structure of) my code to circumvent this OOM error to arise?

Answer 1

There's probably a number of factors here. arrayString is going to need 300MB of memeory. If that's larger than your maximum memory size, that's your problem right there. If your minimum size is smaller, that will slow down the garbage collector as it tries to get the memory that it really does need.

Next, you are repeatedly creating a StringBuilder instance needing roughly 50KB. This space might or might not get recycled efficiently, but it usually takes time for the GC to fully recover larger spaces. So you can be using only 50% or less of the max memory and yet the GC is taking forever to truly free any space. (Not that 50KB is all that large compared to 300MB, but I thought I'd mention it.)

The last problem is that if the GC starts taking too long, it just gives up and throws an OutOfMemoryError. This can happen even when plenty of memory is theoretically available.

The simplest fix is to give it lots more minimum and maximum memory. Don't think that because you're not using the maximum you're safe from an OOM.

It's also worth noting that the rapidity of allocating and freeing memory hurts you. Interleaving some code that makes fewer demands (fewer new's and smaller sizes) can give the GC more time to organize memory. With longer deadlines, the GC won't get so far behind despite the needed work being the same, and hence it won't give up and throw the OOM.

The people who write GCs are always tweaking them and trying for better tradeoffs, so it's a bit tricky to say exactly what the GCs are doing, or what they'll be doing tomorrow. Here, I'm just trying to give you a few ideas.