Sidekiq worker priority setup

Question

Currently am running my background jobs using sidekiq. Concurrently am running 10 jobs at a time. These jobs are not a small jobs. It will take 6-8 hours to complete the work.

In the mean time i need to run some smaller jobs(will take 2-3 mins). Currently am not able to run these jobs until any one of the above 10 jobs to complete. So i need to wait for 6-8 hours to perform this small job.

But these jobs should run immediately after adding to the queue. Is there any option to block any one of the process to run this kind of small jobs. I tried the queue option but its also not working in my scenario.

Here is my sidekiq config

:concurrency: 10
:queues:
  - [web, 7]
  - [default, 3]

Can any one give me the solution for this problem?

Answer 1

The long jobs suffocate the smaller jobs since they fight over the same resources.

I can suggest that you run two sidekiq processes, each using a different config file, and each listening to a different queue:

sidekiq_long.yml:

:concurrency: 10
:queues:
  - default

sidekiq_normal.yml:

:concurrency: 3
:queues:
  - web

This way one sidekiq process will always be available to server short jobs:

sidekiq -C config/sidekiq_long.yml.yml
sidekiq -C config/sidekiq_normal.yml