Service broker - multiple queues vs single queues when communicating with external apps

Question

We currently have an application that uses service broker to queue messages to sent to an external system, which we communicate with via a web service interface.

At present we only integrate with a single company, so one queue is sufficient - however we need to start passing messages to multiple companies who all use the same web service interface.

I'm wondering if a single queue system is sufficient for this or should we have a queue per company. With a queue per company I'm worried about scaling this out as we could have lots of queues and then lots of connections to check the queues.

With a single queue however we can just add more readers as required. However, if we cannot communicate with one of the external systems (e.g. connectivity issue), then there isn't an issue with the message, and we'd want to retry it, but we don't want to delay messages for companies whose systems are up. I was wondering how people are currently dealing with similar scenarios?

We could reinsert the message, but the concern I have about that is that we don't guarantee order of delivery.

Answer 1

I'm assuming that by "reinserting" a message you mean just rolling back the transaction that received it. The effect will be that the message will become available for receive again, as a first message from the given conversation (so you don't need to worry about preserving the order of delivery). That said, there is a problem with this approach, namely the poison message handling. If 5 consecutive receives from a given queue are rolled back, the queue will become disabled.

An alternative approach, described in detail in Klaus Aschenbrenner's book, is to have a table of pending requests. As soon as the activated procedure receives a request message from Service Broker queue, it tries to do the web service call. If that fails for some reason, the request is put in the pending requests table and retried every once a while (you can use conversation timers to schedule retries). That way, if a web service is not available, it won't keep blocking the Service Broker reader from servicing other companies (assuming the timeout for the first request is small enough). And since the receive will be committed no matter if the web service call succeeds, you won't run into poison message problem.