Race Condition in Ruby on Rails

Question

I'm running into a weird bug on Heroku, which I believe may be a race condition, and I'm looking for any sort of advice for solving it.

My application has a model that calls an external API (Twilio, if you're curious) after it's created. In this call, it passes a url to be called back once the third party completes their work. Like this:

class TextMessage < ActiveRecord::Base
   after_create :send_sms

   def send_sms
     call.external.third.party.api(
       :callback => sent_text_message_path(self)
     )
   end
end

Then I have a controller to handle the callback:

class TextMessagesController < ActiveController::Base
  def sent
    @textmessage = TextMessage.find(params[:id])
    @textmessage.sent = true
    @textmessage.save
  end
end

The problem is that the third party is reporting that they're getting a 404 error on the callback because the model hasn't been created yet. Our own logs confirm this:

2014-03-13T18:12:10.291730+00:00 app[web.1]: ActiveRecord::RecordNotFound (Couldn't find TextMessage with id=32)

We checked and the ID is correct. What's even crazier is that we threw in a puts to log when the model is created and this is what we get:

2014-03-13T18:15:22.192733+00:00 app[web.1]: TextMessage created with ID 35.
2014-03-13T18:15:22.192791+00:00 app[web.1]: ActiveRecord::RecordNotFound (Couldn't find TextMessage with id=35)

Notice the timestamps. These things seem to be happening 58 milliseconds apart, so I'm thinking it's a race condition. We're using Heroku (at least for staging) so I think it might be their virtual Postgres databases that are the problem.

Has anyone had this sort of problem before? If so, how did you fix it? Are there any recommendations?

Answer 1

after_create is processed within the database transaction saving the text message. Therefore the callback that hits another controller cannot read the text message. It is not a good idea to have an external call within a database transaction, because the transaction blocks parts of the database for the whole time the slow external request takes.

The simples solution is to replace after_save with after_commit (see: http://apidock.com/rails/ActiveRecord/Transactions/ClassMethods/after_commit)

Since callbacks tend to become hard to understand (and may lead to problems when testing), I would prefer to make the call explicit by calling another method. Perhaps something like this:

# use instead of .save 
def save_and_sent_sms
  save and sent_sms
end

Perhaps you want to sent the sms in the background, so it does not slow down the web request for the user. Search for the gems delayed_job or resque for more information.

Answer 2

Do you have master/slave database where you always write to master but read from slave? This sounds like the db replication lag.

We solved such problems by forcing a read being made from the master database in the specific controller action.

Another way would be to call send_sms after the replication has been finished.