Implementing fault tolerance in distributed message queues

Question

Suppose in the picture below that the middle message queue fails. Senders can still get messages sent by using other message queues.

But what happens if the message queue dies after receiving the message. How does the sender know if the message was sent to the receiver or not to decide whether or not to resend on a different message queue?

Similar what happens if the receiver dies after the message queue delivers its message to it? How would the sender know whether or not its intended request had been fulfilled by the receiver or not?

Answer 1

As a starting point, you need to read http://en.wikipedia.org/wiki/Two_Generals%27_Problem.

This is an instance of a very famous, and very common, problem in computer-science. Technically it is considered 'solved' as we know the answer; however, the short-story is: what you are asking for is (strictly speaking) impossible. There are protocols you can devise that will allow you to achieve any level of confidence the message has (or has not) been delivered, provided that confidence is <1.0.

In practice variations on two and three-phase distributed transaction protocols are used, along with various retransmit and resynchronisation fallbacks. The specifics depend on the implementation.

Often the choice is to permit the possibility of duplicates and require the Receiver to respond appropriately. This is the choice made by TCP, which if you think about it is trying to find a reasonable answer to the same question.