Send File in Chunks with Metadata

Question

I am writing a web app and an OS X app that will upload files to a server.

The web app uses FlowJS to handle sending the files to the server. Along with every request, it sends the chunk number, chunk size, total file size, file name, etc. This is great because behind the scenes I'm uploading this data to an S3 and I use this information to determine when the file is finished uploading. Additionally, since the data is coming in chunks, I don't have to store the entire file in memory on my server.

For OS X with objective c, I'm planning on using AFNetworking. I tried using a multipart upload:

-(void)uploadFileWithUrl:(NSURL*)filePath  {
    AFHTTPRequestOperationManager*manager = [AFHTTPRequestOperationManager manager];
    [manager POST:@"http://www.example.com/files/upload" parameters:nil constructingBodyWithBlock:^(id<AFMultipartFormData> formData) {
        [formData appendPartWithFileURL:filePath name:@"blob" error:nil];
    } success:^(AFHTTPRequestOperation *operation, id responseObject) {
        NSLog(@"Success: %@", responseObject);
    } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
        NSLog(@"Error: %@", error);
    }];
}

but this simply attempts to send the entire file in one request which is not acceptable as the server will run out of memory on large requests.

What I would like to know is if there is anything like FlowJS for objective C, or if there is some way to include that information with AFNetworking for each call.

Answer 1

I do not think that you can do this automatically very easily. (And if one knows how to do this, I would like to know the answer.)

Multipart is no help, because it structures the content, but not the transmission. Think of it as paragraphs in a text document: They give the text a structure, but it is still one file.

That leads to the real problem: You want to have chunks of a fixed size and send a chunk after one is processed (uploaded to S3), then the next chunk and so on. How does the client know that the first chunk is processed? HTTP does not work the way that you can send new data, when the first one received by the server. You send the next chunk, when the first one is sent (on the way). I'm not sure, whether AFNetworking guarantees to request for the next part, after sending the first part … Maybe it wants to collect small parts.

Why don't you do it, S3 does:

Initiate a upload request with an ID as response. Then send a chunk using that upload ID. If it is done, the server responds and the next part is send. Doing so you have a handshake.