pubsub with multiple instances of each consumer

Question

I'm currently looking into setting up a publish subscribe messaging infra structure for our microservice based platform. The new setup is meant to replace our current kafka based one, with something that's a little easier to maintain. One reason that we initially went with kafka, was the ability to have consumer groups. We have multiple instances of each consuming service running and by putting all instances of the same service into the same consumer group, we insured they were spread across partitions, and so, only one of the instances would receive the message(as I understand it).

So, basically, if we have service a, with instances a1 and a2, and service b with b1 and b2, all listening on the same topic, is there any way, with something like rabbitmq or redis, to gaurantee that only one instance of service a and one instance of service receives a message?

Answer 1

Yes, you need two Worker Queues A and B

https://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html

and one Fan Out routing which sends messages to both queues

https://www.rabbitmq.com/tutorials/tutorial-three-dotnet.html

So in rabbitmq you publish to a fan out Exchange and read from persistent queue A or B both of which get a copy of the message from the Exchange.

Now not all messaging systems are equal, and things can get weird if you want to guarentee a message is only read once as most of them have some sort of 'resend if it crashed' mechanic.