Break Out File Download into Background Process Using Sidekiq

Question

I'm trying to break out file download into a background process. My assets are stored on S3.

My original (blocking) code looks like this

# From DownloadsController#download

data = open(path)
send_data(data.read, type: @download.mime_type, filename: @download.file_title)

So I've set up Redis and Sidekiq, and Created a FielDownloadWorker:

class FileDownloadWorker

  include Sidekiq::Worker

  def perform(path, mime_type, file_title)
    data = open(path)
    # What happens next?
  end

end

Which is called using:

FileDownloadWorker.perform_async(path, @download.mime_type, @download.file_title)

How do I initiate download from the worker?

Answer 1

I ended up initiating the file download directly from S3 using Query String Authentication. This way the file is downloaded directly to the client from S3 and the Rails app's thread isn't blocked.

Great writeup here.

Answer 2

You can't. You want the user to receive the file, right? It has to happen within the controller so that the controller can respond with the download. If you're trying to achieve concurrency try using threads:

@data = nil
t = Thread.new { @data = open(path) }
# ... do other stuff ...
t.join # wait for download to finish in other thread
send_data(@data.read, type: @download.mime_type, filename: @download.file_title)

If you decide to go with the worker approach you can have it update a database field or cache when the download is finished, then the user would have to make another request to your app to get the finished file. Something like:

1. User clicks button to initiate download.
2. Page updates with message "Your file is downloading, refresh the page in a few seconds"
3. User refreshes the page, your controller sees that the file is downloaded and does send_data