RabbitMQ High-speed Transient Messaging Performance

Question

The system we are building is receiving data through the external feed. Our job is to distribute this data to multiple services, run the calculations and forward the results elsewhere - typical publisher-subscriber situation. What we need is a very low latency messaging. We don't need to persist the messages like MSMQ.

Is RabbitMq fast enough for a soft realtime message delivery? Are there any benchmarks? Is it a good idea to use it instead of TIBCO Rendezvous? Are there any other open-source soft real time messaging alternatives?

Thanks.

Answer 1

(I am a rabbitmq developer.)

Rabbit will, when lightly loaded, generally have latencies on the order of 100-400 microseconds, depending on things like your network card and CPU speed. Once the loading gets a little heavier, internal buffering starts to appear and latencies rise a little. You can safely expect 1ms latencies until the bandwidth use (messages-per-second, bytes-per-second) starts to get high. Latencies will also rise once persistence is introduced, naturally.

Regarding benchmarks, one of the biggest problems here is defining what's important to your application. There are some trivially simple point-to-point and pub-sub latency-and-throughput measuring examples included with the Java client; ask on the rabbitmq-discuss list if you have problems with them! They don't measure much of relevance to real-world applications, but might help allay any concerns you have regarding microbenchmarks of latency or throughput.

Finally, there are many, many good open-source messaging and messaging-related systems available these days. In the world of AMQP alone, besides RabbitMQ, there are also Qpid and OpenAMQ. There are also good open-source JMS servers out there, if you are able to restrict yourself to Java (many people have success with ActiveMQ). Lots of light-weight systems are springing up for Ruby and Python systems, as well; these systems tend to concentrate on queueing alone, and tend not to have the flexible routing capability that AMQP offers.

Answer 2

You should be able to achieve many tens of thousands of messages per second per CPU. For example one of our standard tests pushes 25k messages per second from a Java client to the server running on a quad core COTS debian box, and back to the client. The client and server are running on the same box, so that's 50k messages processed per second on the server plus 50k messages processed per second on the client. You can get higher rates by running the server on a dedicated box with more cores. For rates based on bytes/second please ask on the rabbitmq-discuss mailing list.

alexis

Answer 3

The best solution I can think about your system is ZeroMQ.

It doesn't have persistance, that you said you don't need and it's VERY quick and simple to use.

It's not an AMQP implementation (that it seems you don't need too) but as it says on this guide:

ØMQ (ZeroMQ, 0MQ, zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry whole messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fanout, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems.