Delay script every X array item in Ruby

Question

I need to throttle my API requests. There is a loop within my script that pulls 1000 values from a csv file and loops through each of them, and each value is a separate API call. If I want to, for example, only make 10 API calls a minute...How could I do that? I'm thinking something like this:

  csv_array.each_slice(10) do |chunk|
        chunk.each do |x|
          make_api_call(x)
          Delay.one_minute
        end
  end

Is there a better option to throttle my API requests, or am I on track with the above thinking? I am using ruby 1.9.3.

Answer 1

I've used something like this in the past:

SLEEP_WINDOW = 60 * 1 # minutes
csv_array.each_slice(10) do |chunk|
  start_time = Time.now
  chunk.each do |x|
    make_api_call(x)
  end
  sleep_time = SLEEP_WINDOW - (Time.now - start_time)
  sleep sleep_time if (sleep_time > 0)
end

The assumption is it's OK to blast through the chunks, then any remaining delay time will case the code to sleep. If the API calls took over a minute the code will loop immediately.

Answer 2

You can use threads to ensure that the correct interval has elapsed before the next call is made. This one spaces out the calls for each member of the csv array:

csv_array.each do |x|
  timer = Thread.new { sleep 6 }
  execution = Thread.new { make_api_call(x) }
  [timer, execution].each(&:join)
end

The benefit here is that it would ensure a minimum of 6 seconds have elapsed, but if the response happens to take longer than that, the timer will already have expired and the next request can begin right away.