celery tasks, workers and queues organization

Question

I have some independent tasks which I am currently putting into different/independent workers.

To be understood easily I will walk you through an example. Let's say I have three independent tasks namely sleep, eat, smile. A task may need to work under different celery configurations. So, I think, it is better to separate each of these tasks into different directories with different workers. Some tasks may be required to work on different servers.

I am planning add some more tasks in the future and each of them will be implemented by different developers.

Providing these conditions, there are more than one workers associated to each individual task.

Now, here is the problem and my question.

When I start three smile tasks, one of these will be fetched by smile's worker and carried out. But the next task will be fetched from eat's worker and never will be carried out.

So, what is the accepted, most common pattern? Should I send each tasks into different queues and workers should listen its own queue?

Answer 1

The answer depends on couple of things that should be taken in consideration:

• Does order of commands should be preserved ?

If so the best approach is placing some sort of command pattern as serialized message so each fetched/consumed message can be executed in it's order in single place in your application. If it's not an issue for you - you can play with topic exchange while publishing different message types in single exchange, and having different workers receiving the messages by predefined pattern. This by the way will let you easily add another task lets say "drink" without changing a line in already existing transportation topology/already existing workers.

• Are you planning scaling queues among different machines to increase throughput ?

In case you have very intense traffic of tasks (in terms of frequency) it may be worth creating different queue for each task type so latter when you grow you can place each one on different node in rabbit cluster.

Answer 2

In a similar setup, I decided to go with specific queues for different tasks, and then I can decide which worker listens on which queue (which can also be changed dynamically !).