MassTransit RabbitMQ Node distribution

Question

I'm investigating using MassTransit with RabbitMQ in our application as an ESB. The main benefit I'm looking for is adding durable asynchronous messaging processing to the incoming data stream.

Our application profile has two parts:

Incoming data stream

• One way messaging
• Processed asynchronously
• 10k+ messages per minute

Website activity

• Two way
• Ideally use c# async await language features but requires data in both directions
• < 100 per minute

The web app messaging isn't a necessity but would be nice to follow the same mechanism to full abstract away data access through the ESB.

Questions:

From what I've read; an ESB node should not know or care about any other node on the bus, it should just do it's own work and send messages onto the bus, waiting for replies if/when required. To me that means each web / app server would have it's own local clustered queue. Is this assumption correct?

If that is correct; how would I programmatically add machines to the cluster? Are there any gotchas I need to be aware of?

If this is not correct; how would I manage the queue cluster? Creating a dedicated cluster has it's own problems such as DNS entries, load-balancing for redundancy / offline nodes, etc

I'm down with the functionality ESBs can add along with MassTransit's implementation however I am a little clouded with the logistics of the best practices of where / how to set it up in a durable configuration.

Thanks for any feedback & advice

Update We are utilising EC2 for machine infrastructure, in particular we use availability zones to minimise any data center outages. With this configuration we have 3 zones, each zone has a web server, app server and db server (Couchbase). We also utilise EC2's load balancers to share load between the zones.

@Travis: Do you have any experience / advice of using MT / RMQ within Amazon's EC2?

Answer 1

So at a scale that's significantly larger than yours, we have a RabbitMQ cluster that sites behind a load balance (an F5). The all processes using MT reference the load balanced address. The only thing each process needs is it's own queue to receive from.

Clustering in RabbitMQ (3.0+) is all handled in the RabbitMQ configuration. The processes/code knows nothing about the clustering.

I'm not sure what you mean by "node" in this question, so it's hard to make sure I'm answering the right question. But as soon as you add a process to the same vhost (default or otherwise) in RabbitMQ, MT will connect all the pieces it needs (exchanges, queues, and bindings).

Answer 2

We run a different approach to Travis. Every machine that has a service that consumes/processes or publishes messages are also nodes in the RabbitMq cluster. Every machine then only has to address RabbitMq via localhost

Each service is on more than one machine To achieve this we are using Competing consumers (i.e. multiple machines will be reading from the same clustered queue (but via localhost)) So your architecture has to allow for parallel processing of messages.

If a machine goes down, there is at least another one that has the dead machines services.

If lots of machines go down, there are other machines with all the messages stored and you can deploy the services to them.