You should follow the Sidekiq documentation and get it setup for Heroku.
One you have Sidekiq running, I recommend the following architecture:
ContactListWorker
: downloads list, enqueues a contact update job for eachContactWorker
: given details, create/update a contact- rake task
contacts:nightly_update
to queue up ContactListWorker job
Here's some rough pseudo code to show what that architecture might look like:
# rake contact:nightly_sync_list
namespace :contacts do
desc "Test Rake Task"
nightly_list_sync: :environment do
ContactListWorker.perform_async
end
end
class ContactListWorker
require 'net/ftp'
include Sidekiq::Worker
def perform()
sftp = Net::SFTP.start('ftp.test.com','luigi', :password => 'pass_word')
records = sftp.download!("luigi/list.xml")
records_hash = Hash.from_xml(records)
records_hash['Report']['Details'].each {|record| ContactWorker.perform_async(record) }
end
end
class ContactWorker
include Sidekiq::Worker
def perform(record)
contact = Contact.create(
first_name: record['FirstName'],
last_name: record['LastName'],
date_of_birth: record['DateofBirth']
)
if contact.valid?
puts "Created contact"
else
puts "Invalid contact"
end
end
end
This architecture allows you to asynchronously kick off one background ContactListWorker
job. This job will do the download and quickly enqueue N ContactWorker
jobs asynchronously. This allows you to fan-out the processing across multiple Sidekiq workers and threads to distribute the processing.