Question

Say I have a queue that has high load throughout the day. Each of the messages in the queue takes several seconds to process.

If I have 4 machines all setup with external activator (all configured for the same queue/service), will that work?

If it does work, will it load balance (spreading the work out evenly across the worker machines)?

Was it helpful?

Solution

This boils down, ultimately, to how multiple pending WAITFOR (RECEIVE) statements are serviced. Consider you have 4 instances of your app running, each 'listens' on the same queue, by issuing a WAITFOR(RECEIVE). A message becomes available in the queue, the question is: which of the pending WAITFOR(RECEIVE) will get the message?

The answers surprises many, but this is it and is by design: the most recently issued WAITFOR(RECEIVE) will get the message. Not the oldest one, not a random one, but the newest one. In other words, LIFO order for the listeners.

Why? This behavior is intentional and the intent is to have as few listeners as needed to handle the load. You originally have 4 processes listening. One gets the message, processes it and the it issues again the WAITFOR(RECEIVE), thus starting to listen again for more messages. The next message, when it arrives, will be given, deterministically, to the same process, because is the most recently issued WAITFOR(RECEIVE). This listener will get the message, process it and then issue again WAITFOR(RECEIVE). And will, again, deterministically get the next message. Here is the gist of it: if this single listener can handle all the incoming traffic, the other 3 are redundant. After a while their pending WAITFOR(RECEIVE) will time out and these processes can safely go away (exit).

A second process may get a message only if it arrives (becomes available) while the first listener is busy processing another message (ie it had not posted a new WAITFOR(RECEIVE)).

So why is this good? There is another piece of the puzzle: Understanding When Activation Occurs. If the lone processing thread can no longer handle the incoming traffic, a new processing thread (internal or external) will be spawned, up to the configured MAX_QUEUE_READERS.

So there you have it, between these two behaviors you get a self-throttled optimal number of readers (processes, threads):

  • when traffic spikes and there are too few reader the activation mechanism will spawn a new processing thread. This is triggered when 5 seconds pass an no processing thread manages to 'drain' the queue (RECEIVE returns empty rowset)
  • when traffic slows down and there are too many listeners activated the minimum required to handle the traffic is kept alive, while the surplus is left to time out
  • if the traffic dries completely for a period, all threads/processes may go away (0 pending WAITFOR(RECEIVE)). When a message arrives a processing thread will be activated to process it.

Keep in mind that due to correlated message locking (Conversation Group Locks) a queue may have messages in it, but none of them be available for process by a new thread (ie. they're all locked). For activation/waitfor purposes this situation means the queue is 'empty' (no messages are available for a new processor).

Finally, for all this to happen, the application has to behave correctly:

  • activate a new thread when notified (internal activation does this, the external activator also does it correctly)
  • the processing threads should issue a WAITFOR(RECEIVE) for a reasonable time out and exit if the statement times out (it will return an empty result set, not error).
  • it is imperative to issue a RECEIVE when activated. If you fail to do so, the queue monitor will go into NOTIFIED state and won't activate anything again. See Understanding Queue Monitors.

OTHER TIPS

To utilise the built-in load balancing you would need to deploy the service to more than one sql server instance. I suspect that isn't quite what you are planning, so you will have to come up with a custom method, such as having an internal activated procedure that forwards your arriving messages into one of your four queues that the external activation processes look at.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top