REST: Uploading large files

Question

I am wondering what is the recommended way to upload a large file using REST.

Giving that the time taken from the server to write the file on disk can be large, should I wait for the writing operation to end before sending the 201 reply to the client?

Is it maybe better to reply to the user on the fly and process the writing operation later? In this second case what http status should i return? In fact even though i return a 201 no one can ensure that the writing operation was successful.

Answer 1

It sounds as if you're not starting to write to disk until you've read the last byte from HTTP. That has two undesirable effects:

1. You're putting a load of data into your server's memory footprint
2. You're pushing all the disk IO to the end of the request handling, causing a pause that looks bad from the client perspective. In extreme cases, the client may time-out.

Instead, you need to be streaming. As soon as you have some data from the client, start writing to disk. When the client sends its final chunk, you only need to handle that small chunk before sending the 201 response.

If the API you're using doesn't allow you to consume HTTP entities in a streaming manner, I'm afraid you need to look for a different API.

For example, in the Java Servlet API, you can get an InputStream for the request content:

InputStream content = request.getInputStream();

// do something better than this
OutputStream out = new FileOutputStream("content.txt");
byte[] buffer = new byte[1024];
int len;
while ((len = in.read(buffer)) != -1) {
    // whatever processing you want here
    out.write(buffer, 0, len);
}

// now you can send the 201

It's important to understand that when you get the InputStream in this example, your server hasn't started reading the content off the TCP stack yet (*). Only the headers have been read. If it's a large file, it's very likely that the client is still sending at this moment. You read a chunk, write a chunk, in a loop.

---

(*) That's a simplification. It may be in a buffer. But conceptually, you can ignore that