Read a file(>150MB) and return the file content as ByteArrayOutputStream

Question

I am trying to read a large file (>150MB) and return the file content as a ByteArrayOutputStream. This is my code...

private ByteArrayOutputStream readfileContent(String url) throws IOException{

    log.info("Entering readfileContent ");
    ByteArrayOutputStream writer=null;
    FileInputStream reader=null;

    try{
        reader = new FileInputStream(url);
        writer = new ByteArrayOutputStream();

        byte[] buffer = new byte[1024];

        int bytesRead = reader.read(buffer);
        while (bytesRead =  > -1) { 
            writer.write(buffer, 0, bytesRead);
            buffer = new byte[1024];
        }

    }
    finally {
        writer.close();
    }

    log.info("Exiting readfileContent ");
    return writer;
}

I am getting an java.lang.OutOfMemoryError: Java heap space exception. I have tried increasing the java heap size, but it still happens. Could someone please assist with this problem.

Answer 1

Your approach is going to use at least the same ammount of memory as the file, but because ByteArrayOutputStream is using a byte array as storage, it'll potentially have to resize itself 150,000 times (150 meg/1024k buffer) which is not efficient. Upping the heap size to 2* your file size and increasing the size of buf to something much larger may allow it to run, but as other posters have said, it's far better to read form the file as you go, rather than read it in as a String.

Answer 2

You should return the BufferedInputStream and let the caller read from it. What you are doing is copying the whole file into memory as a ByteArrayOutputStream.

Your question is missing what you want to do with the file content. Without that we can only guessing. There is a ServletOutputStream commented out. Did you want to write to this originally? Writing to this instead to the ByteArrayOutputStream should be working.

Answer 3

There is an error in the while loop. Change it to

 while (bytesRead >= -1) { 
     writer.write(buffer, 0, bytesRead);
     bytesRead = reader.read(buffer);
 }

Also don't forget to close reader.

(It will still need quite large amount of memory.)

Answer 4

Since you know how many bytes you are going to be read, you can save time and space by creating the ByteArrayOutputStream with a size. This will save the time and space overheads of "growing" the ByteArrayOutputStream backing storage. (I haven't looked at the code, but it is probably using the same strategy as StringBuilder; i.e. doubling the allocation each time it runs out. That strategy may end up using up to 3 times the file size at peak usage.)

(And frankly, putting the output into a ByteArrayOutputStream when you know the size seems somewhat pointless. Just allocate byte array big enough and read directly into that.)

Apart from that, the answer is that you need to make the heap bigger.

Answer 5

I have seen similar issues in C# in Windows cause by not having enough Contiguous Virtual Memory on the host. If your on Windows, you can try increasing the VM space.