How to add a rate limit to a Resque queue?

Question

So I have a Resque worker that calls an API, the issue is the API has a rate limit of 2 requests per second.

Is there a way to add a delay between each job processed in a specific queue?

P.S. the queue could have thousands of pending jobs.

Answer 1

Per my comment, I'd recommend just performing a sleep for a given number of seconds at the end of each Resque process method.

Answer 2

Why not sleep for a given amount of time at the end of the process? Well, perhaps you want your resque worker to be doing something useful instead. In CPU time, half a second is a lot of time - you could have done something useful there, like process a job from another queue that's not rate limited.

I have this same problem myself, so I'm motivated to find a solution. It seems like there are two easy-ish ways to do it. The first idea is to use resque scheduler and pre-compute the time to run the job at before inserting it. This seems error-prone to me. The second is to use a gem like https://github.com/flyerhzm/resque-restriction (disclaimer: just found it through some googling. haven't used it yet) and rate-limit as you pull jobs off the queue. Seems like a robust solution in theory. Note that if you can't execute the job yet, it never comes off the queue, so you'll pull something else instead - much more efficient use of your workers.