How can I upload images in bulk (about 100-200 images at a time) in Ruby on Rails?

Question

I want to upload images (around 200 kB each) in bulk. We have multiple options like as CarrierWave, Paperclip, and others. How can I perform these uploads in bulk?

Answer 1

TL;DR

Don't. Rails isn't optimized for bulk uploads, so do it out-of-band whenever you can.

Use FTP/SFTP

The best way to deal with large volumes of files is to use an entirely out-of-band process rather than tying up your Rails process. For example, use FTP, FTPS, SCP, or SFTP to upload your files in bulk. Once the files are on a server, post-process them with a cron job, or use inotify to kick off a rake task.

NB: Make sure you pay attention to file-locking when you use this technique.

Use a Queue

If you insist on doing it via Rails, don't upload hundreds of files. Instead, upload a single archive containing your files to be processed by a background job or queue. There are many alternatives here, including Sidekiq and RabbitMQ among others.

Once the archive is uploaded and the queued job submitted, your queue process can unpack the archive and do whatever else needs to be done. This type of solution scales very well.

Answer 2

Like other things in computer Sc, the answer is it depends™. What I really mean is

1. End-Users going to be uploading these?. If yes use jQuery File Upload plugin to present an easy to use interface.
2. For storage, you can storage it on your server. or even better, upload the images directly from users computers to amazon s3. here is an example of Uploading Files to S3 in Ruby with Paperclip
3. Obviously you will need to convert this into background job where all images get images with ajax in separate jobs. if you already don't have a fav que system, I would suggest resque or sidekiq

Note: if you choose to upload images directly to S3 via CORS, then your rails server is free need from managing file uploads. And direct uploads are suggested if you have large images or large number of files to be uploaded.

However, direct uploads limit your ability to modify the images(resize etc). so keep that in mind if you are choosing the direct upload solution.