Upload with paperclip very slow (unicorn)

Question

Sitting here with a simple rails 3 app in which I have a simple Gallery model and each gallery has many images. The image model is extended with paperclip and with the following options

has_attached_file :local, 
   :styles => {
     :large => "800x800>", 
     :medium => "300x300>", 
     :thumb => "100x100#", 
     :small => "60x60#"
   }

In my galleries_controller I have the following action that is implemented in order to work with the jQuery-File-Upload plugin. thereby the json response.

def add_image
   gallery = Gallery.find params[:id]
   image = gallery.images.new({:local => params[:local]})
   if image.save
     render :json => {:thumb => image.url(:thumb), :original => image.url}
   else  
    render :json => { :result => 'error'}
   end
end

To me this is fairly straight forward. But here comes the issue. In Development under mongrel any kind of upload works just fine with about 500-1000ms/upload.

However when I push it in to production I constantly get timeouts of my unicorn workers and when it does send an image through it takes anywhere from 30-55 seconds for one file.

the files I upload are around 100k in size

I have done some testing of the bandwidth between my VPS and my dev computer witH ipref and got an average speed of about 77kbps so the upload should not be a problem.

Note I also did a test with a non ajax file upload using the same app with user model that has an avatar. Development => Completed 302 Found in 693ms Production => Completed 302 Found in 21618ms

Anyone experienced a similar issue with (rails3, unicorn) file uploads?

Answer 1

So after digging around I managed to determine that on my VPS it was the OpenMP Option in ImageMagick that was causing the very slow operation. So my first attempt was to rebuild the native Ubuntu 10.04 package with the --disable-openmp flag added. This failed for some reason and while I am not sure why the package kept comming out with openMP still active. My current solution is now instead to backport ImageMagick from Ubuntu 10.10. Below follows the steps I took:

Step 1 download the following files:

• imagemagick_6.6.2.6-1ubuntu1.1.dsc
• imagemagick_6.6.2.6.orig.tar.bz2
• imagemagick_6.6.2.6-1ubuntu1.1.debian.tar.bz2

from here

Step 2 unpack the package

$ dpkg-source -x imagemagick_6.6.2.6-1ubuntu1.1.dsc

Step 3 edit the rules

$ cd imagemagick-6.6.2.6
$ vim debian/rules

Add the the follwing line to the ./configure statment on line 25-39. I added mine on line 34.

34: --disable-openmp \

Step 4 add dependencies and build ( I needed these dependencies)

$ sudo apt-get install liblqr-1-0-dev librsvg2-dev
$ dpkg-buildpackage -b

Step 5 Out with the old, in with the new

$ sudo apt-get remove --purge imagemagick
$ sudo dpkg -i libmagickcore3_6.6.2.6-1ubuntu1.1_amd64.deb
$ sudo dpkg -i libmagickwand3_6.6.2.6-1ubuntu1.1_amd64.deb
$ sudo dpkg -i imagemagick_6.6.2.6-1ubuntu1.1_amd64.deb

Step 6 Once again have fast image conversions

_before_ (with openmp)
$ time utilities/convert 'image.jpg' -resize "x60" -crop "60x60+10+0" +repage 'thumb'
real    0m11.602s
user    0m11.414s
sys  0m0.069s

_after_
$ time utilities/convert 'image.jpg' -resize "x60" -crop "60x60+10+0" +repage 'thumb'
real    0m0.077s
user    0m0.058s
sys  0m0.019s

Answer 2

If processing takes a long time, consider processing the thumbnails in a separate workers.

Request: accept file; save it to disk; post job to queue Worker: pop job from queue; create thumbnails; repeat

Delayed::Job and Resque are great solutions for this.