Importing huge excel file to Rails application

Question

I have an excel file with thousands of rows. In my case, I can't use bulk insert, because for each row I should create few associations. Now, all process take more than 1 hour with 20k rows, which is hell. What is the best way to resolve this problem?

I'm using spreadsheet gem.

Answer 1

This is analogous to the infamous "1+N" query situation that Rails loves to encounter. I have a similar situation (importing files of 20k+ rows with multiple associations). The way I optimized this process was to pre-load hashes for the associations. So for example, if you have an AssociatedModel that contains a lookup_column that is in your import data, you would first build a hash:

associated_model_hash = Hash.new(:not_found)

AssociatedModel.each do |item|
  associated_model_hash[item.lookup_column] = item
end

This provides a hash of objects. You can repeat for as many associations as you have. In your import loop:

associated_model = associated_model_hash[row[:lookup_column]]
new_item.associated_model_id = associated_model.id

Because you don't have to do a search on the database each time, this is much faster. It should also allow you to use bulk insert (assuming you can guarantee that the associated models will not be deleted or modified in a bad way during the load).