How can I pipe a rack file upload directly to S3?

Question

Is it possible to pipe a file upload using ruby?

Instead of reading the whole file into memory, or writing a temp file to then write to S3, I was hoping to stream the request body directly.

Streaming data to S3 with ruby shows how to stream to S3, but I'm not sure how to turn the request body in a rack app into a readable stream.

class FastUploadApp
  def self.call(env)
    # what goes here?
  end
end

Answer 1

You're exactly right that you need to turn the request body into a readable stream. Specifically, S3 expects a Ruby IO class (in that it wants a #read method and a #eof? method). Rack request bodies don't have #eof? defined, however, so you have to make a little wrapper class:

class RackS3Wrapper
  def initialize(body)
    @body = body
    @eof = false
  end

  def read(*args)
    ret = @body.read(*args)

    if ret == nil or ret == ""
      @eof = true
    end

    ret
  end

  def eof?
    @eof
  end
end

Then you can use this wrapper to stream the request to S3 directly:

s3.buckets['com.mydomain.mybucket'].objects['filename'].write(
  :data => RackS3Wrapper.new(request.body), 
  :content_length => request.env['CONTENT_LENGTH'].to_i)

This hasn't been tested in production or anything, but it should work fine.

Answer 2

Check out my project middl: https://github.com/diclophis/middl

It has some caveats ... mostly you have to rely on the client sending an appropriate request size header (which is not required by the w3 spec)