Rails+Dropbox Sync Architecture

Question

Rails 4, Ruby 2

I have a Rails app that syncs the user's data with Dropbox. Here's how the sync currently flows:

1. Which records in MySQL have their updated_flag set and need to be synced?
2. Okay, we now have a list of which database records need to be synced.
3. Let's iterate through this list and make Dropbox API calls. We upload/merge these records one-by-one.
4. <Data Crunching Sound>
5. Okay, reset the updated_flag for each record once it has been synced with Dropbox.
6. Done!

This works great except for when the user is new and imports a bunch of data (thousands of records). This process is super slow--so slow that Unicorn times out and kills the process.

If you're familiar with Day One, I'm using a very similar data architecture except now in a web app.

So here is my question: Even though I want this sync to happen immediately when the user clicks "Sync", should this whole operation be put in the background using something like Delayed Job? Or is there a more efficient way to process these so that a single HTTP request to sync doesn't time out because it's cranking through hundreds of queries?

Answer 1

Well, since no one else is chiming in, I'll give what I've since learned as the answer.

Yes, a background process is needed to do these. Some sync jobs could involve hundreds of queries, and they currently get initiated with a single HTTP request. They won't finish in time before unicorn times out (60 seconds) so Delayed Job (or similar) is needed here.