Is Sidekiq suitable for tasks that are highly mission critical, one-time execution guarrantee required, single threaded execution necessary?

Question

Example Scenario:

Payment handling and electronic-product delivery transaction.

Requirements

1. There are approximately a few thousand payment transactions a day that need to be executed. Each taking about 1 second. (So the entire process should take about an hour)
2. Transactions must be processed linearly in a single thread (the next transaction must not start until the last transaction has completed, strong FIFO order is necessary)
3. Each payment transaction is wrapped inside a database transaction, anything that goes back to roll the transaction back, it is aborted and put into another queue for manual error handling. After that, it should continue to process the rest of the transactions.

Order of Importance

1. Single execution (if failed, put into error queue for manual handling)
2. Single Threadedness
3. FIFO

Is Sidekiq suitable for such mission critical processes? Would sidekiq would be able to fullfil all of these requirements? Or would you recommend other alternatives? Could you point me to some best practices regarding payment handling in rails?

Note: The question is not regarding whether to use stripe or ActiveMerchant for payment handling. It is more about the safest way to programmatically execute those processes in the background.

Answer 1

Yes, Sidekiq can fulfill all of these requirements.

To process your transactions one at a time in serial, you can launch a single Sidekiq process with a concurrency of 1 that only works on that queue. The process will work jobs off the queue one at a time, in order.

For a failed task to go into a failure queue, you'll need to use the Sidekiq Failures gem and ensure retries are turned off for that task.

To guarantee that each task is executed at least once, you can purchase Sidekiq Pro and use Reliable Fetch. If Sidekiq crashes, it will execute the task when it starts back up. This assumes you will set up monitoring to ensure the Sidekiq process stays running. You might also want to make your task idempotent, so it doesn't write the same transaction twice. (In theory, the process could crash after your database transaction commits, but before Sidekiq reports to Redis that the task completed.)

If using Ruby is a constraint, Sidekiq is probably your best bet. I've used several different queuing systems in Ruby and none of them have the reliability guarantee that Sidekiq does.

Answer 2

the next transaction must not start until the last transaction has completed

In that case, I think background processing is suitable as long as you create the next job at the completion of the previous job.

class Worker
  include Sidekiq::Worker

  def perform(*params)
    # do work, raising exception if necessary

    NextWorker.perform_async(params, here)
  end
end