Big difference in throughput when using MSMQ as subscription storage for Nservicebus as opposed to RavenDB

Question

Recently we noticed that our Nservicebus subscribers were not able to handle the increasing load. We have a fairly constant input stream of events (measurement data from embedded devices), so it is very important that the throughput follows the input.

After some profiling, we concluded that it was not the handling of the events that was taking a lot of time, but rather the NServiceBus process of retrieving and publishing events. To try to get a better idea of what goes on, I recreated the Pub/Sub sample (http://particular.net/articles/nservicebus-step-by-step-publish-subscribe-communication-code-first).

On my laptop, using all the NServiceBus defaults, the maximum throughput of the Ordering.Server is about 10 events/second. The only thing it does is

class PlaceOrderHandler : IHandleMessages<PlaceOrder>
{
    public IBus Bus { get; set; }

    public void Handle(PlaceOrder message)
    {
        Bus.Publish<OrderPlaced>
            (e => { e.Id = message.Id; e.Product = message.Product; });
    }
}

I then started to play around with configuration settings. None seem to have any impact on this (very low) performance:

Configure.With()
           .DefaultBuilder()
           .UseTransport<Msmq>()
           .MsmqSubscriptionStorage();

With this configuration, the throughput instantly went up to 60 messages/sec.

I have two questions:

1. When using MSMQ as subscription storage, the performance is much better than RavenDB. Why does something as trivial as the storage for subscription data have such an impact?

2. I would have expected a much higher performance. Are there any other configuration settings that I should use to get at least one order of magnitude better than this? On our servers, the maximum throughput when running this sample is about 200 msg/s. This is far from spectacular for a system that doesn't even do anything useful yet.

Answer 1

MSMQ doesn't have native pub/sub capabilities so NServiceBus adds support this by storing the list of subscribers and then looping over that list sending a copy of the event to each of the subscribers. This translates to X message queuing operations where X is the number of subscribers. This explains why RabbitMQ is faster since it has native pub/sub so you would only need one operation against the broker.

The reason the storage based on a msmq queue is faster is that it's a local storage (can't be used if you need to scaleout the endpoint) and that means that we can cache the data since that can't be any other endpoint instances updating the storage. In short this means that we get away with a in memory lookup which as you can see is the fastest option.

There are plans to add native caching across all storages:

https://github.com/Particular/NServiceBus/issues/1320

200 msg/s sounds quite low, what number do you get if you skip the bus.Publish? (just to get a base line)

Answer 2

Possibility 1: distributed transactions Distributed transactions are created when processing messages because of the combination Queue-Database. Try measuring without transactional handling of the messages. How does that compare?

Possibility 2: msmq might not be the best queueing system for your needs Ever considered switching to rabbitmq for transport? I have very good experiences with RabbitMq in combination with MassTransit. Way exceed the numbers you are mentioning in your question.