BufferedInputStream into byte[] to be send over a Socket to a Database

Question

I have been looking around for an answer to this, but couldn't really find anything on it. Earlier today, I asked how I could make a File into a String through a byte array, and then back again, for retrieval later.

What people told me, was that I had to just store the byte array, to avoid nasty encoding issues. So now I've started working on that, but I have now hit a wall.

Basically, I used unbuffered streams before, to turn a file into a byte array. This works good in theory, but it takes up a lot of memory which eventually will cast the heap size exception. I should use buffered streams instead (or so I am told), and the problem I have now, is going from a BufferedInputStream to a byte[]. I've tried to copy and use the methods found in this documentation

http://docs.guava-libraries.googlecode.com/git/javadoc/index.html?com/google/common/io/package-summary.html

Where I exchange unbuffered streams for buffered streams. The only issue, is that I can't directly turn a buffered output stream into a byte array, as I can with an unbuffered stream.

Help? :)

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

public final class BufferedByteStream {

    private static final int BUF_SIZE = 1024000;

    public static long copy(BufferedInputStream from, BufferedOutputStream to) throws IOException { 
        byte[] buf = new byte[BUF_SIZE];
        long total = 0;
        while(true) {
            int r = from.read(buf);
            if(r == -1) {
                break;
            }
            to.write(buf, 0, r);
            total += r;
        }
        return total;
    }

    public static byte[] toByteArray(BufferedInputStream in) throws IOException {
        BufferedOutputStream out = new BufferedOutputStream(new ByteArrayOutputStream());
        copy(in, out);
        return out. // <--- Problem is here
    }
}

EDIT:

I am still getting Heap Space errors. So I will now post all the code:

main.java

import java.io.*;
import java.util.logging.Level;
import java.util.logging.Logger;
import jserver.io.BufferedByteStream;
/**
*
* @author Vipar
*/
public class main {
    public static void main(String[] args) {
    File f = new File("<doesn't matter>");
        try {
            byte[] buf;
            try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream(f))) {
                buf = BufferedByteStream.toByteArray(bis);
                bis.close();
            }
            File f2 = new File("<doesn't matter>");
            try (FileOutputStream fos = new FileOutputStream(f2)) {
                fos.write(buf);
                fos.close();
            }
        } catch (FileNotFoundException ex) {
            Logger.getLogger(main.class.getName()).log(Level.SEVERE, null, ex);
        } catch (IOException ex) {
            Logger.getLogger(main.class.getName()).log(Level.SEVERE, null, ex);
        }
    }
}

BufferedByteStream.java

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

public final class BufferedByteStream {

    private static final int BUF_SIZE = 1024000;

    public static long copy(BufferedInputStream from, BufferedOutputStream to) throws IOException { 
        byte[] buf = new byte[BUF_SIZE];
        long total = 0;
        while(true) {
            int r = from.read(buf);
            if(r == -1) {
                break;
            }
            to.write(buf, 0, r);
            total += r;
        }
        return total;
    }

    public static byte[] toByteArray(BufferedInputStream in) throws IOException {
        ByteArrayOutputStream bytesOut = new ByteArrayOutputStream();
        BufferedOutputStream out = new BufferedOutputStream(bytesOut);
        copy(in, out);
        return bytesOut.toByteArray();
    }
}

Answer 1

Have a look at ByteArrayOutputStream: Java 7 API java.io.ByteArrayOutputStream

bytesOut = new ByteArrayOutputStream();
byte[] bytes = bytesOut.toByteArray();

Update: If you insist on doing what you are doing you can just assign the intermediate ByteArrayOutputStream to a variable and get hold of the array that way:

ByteArrayOutputStream bytesOut = new ByteArrayOutputStream()
BufferedOutputStream out = new BufferedOutputStream(bytesOut);
copy(in, out);
return bytesOut.toByteArray();

Update 2: The real question seems to be how to copy a file without reading it all into memory first:

1) Manually:

    byte[] buff = new byte[64*1024]; //or some size, can try out different sizes for performance
    BufferedInputStream in = new BufferedInputStream(new FileInputStream("fromFile"));
    BufferedOutputStream out = new BufferedOutputStream(new FileoutputStream("toFile"));
    int n = 0;
    while ((n = in.read(buff)) >= 0) {
        out.write(buff, 0, n);
    }
    in.close();
    out.close();

2) Efficiently by the OS and no loop etc:

FileChannel from = new FileInputStream(sourceFile).getChannel();
FileChanngel to = new FileOutputStream(destFile).getChannel();
to.transferFrom(from, 0, from.size());
//or from.transferTo(0, from.size(), to);
from.close();
to.close();

3) If you have Java 7 you can simplify exception and stream closing or just copy the file with the new APIs i in java 7:

java.nio.file.Files.copy(...);

see java.nio.file.Files

Answer 2

DataFetcher would be perfect for this:

http://tus.svn.sourceforge.net/viewvc/tus/tjacobs/io/DataFetcher.java?revision=34&view=markup

You can use clearBuffer to clear out the buffer between reads if you are running out of memory

You'll also need the Timeout class - it's in the same project in the same package.