nodejs - upload using fs.createReadStream, requestjs, streaming and AWS S3 presigned url throws error

Question

My Environment: Windows 8.1, NodeJS 0.10.0 request 2.30.0

This is what I tried based on samples from Amazon SDK and documentation for requestjs

var params = {Bucket:'myprivateuniquebucket'
    ,Key:'mykey'
    ,Expires: 60};
var url = s3.getSignedUrl('putObject', params);
fs.createReadStream('pic.png').pipe(request.put(url));

and this is the error I got

events.js:72
    throw er; // Unhandled 'error' event
          ^
Error: read ECONNRESET
at errnoException (net.js:863:11)
at TCP.onread (net.js:524:19)

However instead of using fs.createReadStream when I used fs.readFile like below everything worked just fine

    fs.readFile('pic.png', function (err, data) {
        request(
            { method: 'PUT'
                , uri: url
                , body: data
            }
            , function (error, response, body) {
            }
        );
    });

Note I followed the streaming example here https://github.com/mikeal/request almost verbatim.

I'd appreciate pointers on what was incorrect since I'd like to use streaming in multiple places.

Answer 1

This is mostly an issue with Amazon's API rather than your code. Uploading a file to Amazon requires the Content-Length header to be set. In your stream example, the length is unknown so Amazon is just dropping the connection. In your second example, request knows the length of the buffer and sets the Content-Length for you.

If you are streaming because the file is huge and loading it all into RAM is hard, then you will need to use Amazon's multi-part uploading APIs. If you are streaming but the file is small, then I'd probably stick with the readFile method.