MSMQ, WCF and robustness

Question

Not being an expert on MSMQ or WCF, I have read up a fair bit about it and it sounds and looks great. I am trying to develop something, eventually but first some theory, which needs to be robust and durable.

MSMQ I guess will be hosted on a seperate server.

There will be 2 WCF services. One for incoming messages and the other for outgoing messages (takes a message, does some internal processing/validation then places it on the outgoing messages queue or maybe sending an email/text message/whatever)

I understand with the right configuration, we can have the system so that it can be transactional (no messages are ever lost) and can be sent exactly once, so no chance of duplication of messages.

The applications/services will be multithreaded to process messages, which there will be hundreds and thousands of them.

BUT during the processing of a message or through the services lifetime, what if the server crashes? What if the server reboots? What if the service throws an exception for whatever reason? How is it possible to not lose that message but some how to put it back on the queue waiting for it to be processed again? Also how is it possible to make sure that the service is robust in such a way that it will spawn itself again?

I'd appreciate any advice and details here. There is quite alot to take in and WCF/MSMQ exposes quite alot of options.

Answer 1

Your assumption:

MSMQ I guess will be hosted on a seperate server.

is incorrect. MSMQ is installed on all machines which want to participate in message queuing.

There will be 2 WCF services. One for incoming messages and the other for outgoing messages

In the most typical configuration, the destination queues are local to the listening service.

For example, your ServiceA would have a local queue from which it reads. ServiceB also has a local queue from which it reads. If ServiceA wants to call ServiceB it will put a message into ServiceB's local queue.

I understand with the right configuration, we can have the system so that it can be transactional (no messages are ever lost)

This is correct. This is because MSMQ uses a messaging pattern called store-and-forward. See here for an explanation.

Essentially the reason it is safe to assume no message loss is because the transmission of a message from one machine to another actually takes place under three distinct transactions.

1. The first transaction: ServiceA writes to it's own temporary local queue. If this fails the transaction rolls back and ServiceA can handle the exception.
2. Second transaction: Queue manager on ServiceA machine transmits message to Queue manager on ServiceB machine. If failure then message remains on temporary queue.
3. Third transaction: ServiceB reads the message off local queue. If ServiceB message handler method throws exception then transaction rolls message back to local queue.

The applications/services will be multithreaded to process messages

This is fine except if you require order to be preserved in the message processing chain. If you need ordered processing then you cannot have multiple threads without implementing a re-sequencer to reapply order.

I thought that MSMQ can be hosted seperately and have x servers share that queue?

All servers which want to participate in the exchange of messages have MSMQ installed. Each server can then write to any queue on any other server.

The reason for my thinking was because what if the server goes down? Then how will the messages get sent/received into MSMQ

If the queues are transactional then that means messages on them are persisted to disk. If the server goes down then when it comes back up the messages are still there. While a server is down it obviously cannot participate in the exchange of messages. However, messages can still be "sent" to that server - they just remain local to the sender (in a temporary queue) until the destination server comes back on-line.

so by having one central MSMQ server (and having it mirrored/failover) then there will be guarentee of uptime

The whole point of using message queueing is it's a fault-tolerant transport, so you don't need to guarantee uptime. If you have a 100% availability then there would be little reason to use message queuing.

how will WCF be notified of messages that are incoming?

Each service will listen on its own local queue. When a message arrives, the WCF runtime causes the handling method to be called and the message to be handled.

how will the service be notified of failures of sending messages

If ServiceA fails to transmit a message to ServiceB then ServiceB will never be notified of that failure. Nor should it be. ServiceA will handle the failure to transmit, not ServiceB. Your expectation in this instance creates a hard coupling between the services, something which message queueing is supposed to remove.

Answer 2

MSMQ can store messages even if temporary shutdown the service or reboot computer.
Main goal of WCF is transport message from source to destination. Doesn't matter what is the transport. In your case MSMQ is transport for WCF and not obvious to have online / available both client and service simultaneously. But when message is received, it's your responsibility to correctly process it, despite what transport was used to send message.