Questions Concerning Using Celery with Multiple Load-Balanced Django Application Servers

StackOverflow https://stackoverflow.com/questions/10098267

  •  30-05-2021
  •  | 
  •  

Question

I'm interested in using Celery for an app I'm working on. It all seems pretty straight forward, but I'm a little confused about what I need to do if I have multiple load balanced application servers. All of the documentation assumes that the broker will be on the same server as the application. Currently, all of my application servers sit behind an Amazon ELB and tasks need to be able to come from any one of them.

This is what I assume I need to do:

  • Run a broker server on a separate instance
  • Configure each application instance to connect to that broker server
  • Each application instance will also be be a celery working (running celeryd)?

My only beef with that is: What happens if my broker instance dies? Can I run 2 broker instances some how so I'm safe if one goes under?

Any tips or information on what to do in a setup like mine would be greatly appreciated. I'm sure I'm missing something or not understanding something.

Was it helpful?

Solution

You are correct in all points.

How to make reliable broker: make clustered rabbitmq installation, as described here: http://www.rabbitmq.com/clustering.html

OTHER TIPS

For future reference, for those who do prefer to stick with RabbitMQ...

You can create a RabbitMQ cluster from 2 or more instances. Add those instances to your ELB and point your celeryd workers at the ELB. Just make sure you connect the right ports and you should be all set. Don't forget to allow your RabbitMQ machines to talk among themselves to run the cluster. This works very well for me in production.

One exception here: if you need to schedule tasks, you need a celerybeat process. For some reason, I wasn't able to connect the celerybeat to the ELB and had to connect it to one of the instances directly. I opened an issue about it and it is supposed to be resolved (didn't test it yet). Keep in mind that celerybeat by itself can only exist once, so that's already a single point of failure.

Celery beat also doesn't have to be a single point of failure if you run it on every worker node with:

https://github.com/ybrs/single-beat

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top