MSMQ: Messages occasionally not being sent or received without error

Question

I have the following setup and problem with MSMQ. Based on previous experience with MSMQ I'm betting that it is something simple I'm missing but I just don't know what it is.

The Setup

I have 3 load-balanced web servers (lets call them Servers W1, W2 and W3) and 1 server which processes certain events/data away from web requests (which I'll call P). All 3 of the web servers, once a particular event occurs within the web application, will send a message to a remote private queue on Server P, which will then process each message from the queue and carry out some task.

The Problem

For the most part - at a guess 95% of the time - everything runs fine, but occasionally Server P does not receive messages from the web servers. This is either because W1, W2 or W3 are not sending them or they are not being received by P, I just can't tell. This means I'm missing vital events happening from the users on the web application but I cannot find any errors listed in my own logs.

The Details

Here are all the details I can think of which may help explain my setup and what I've figured out so far:

• The private queue on Server P is non-transactional.
• The private queue has permissions setup for Everyone to both Send and Receive Messages.

• This is the code I use (C#) to send the message to the remote private queue:

  var queue = new MessageQueue(@"FormatName:DIRECT=OS:ServerP\PRIVATE$\MyMessageQueue");
  
  var defaultProperties = queue.DefaultPropertiesToSend;
  defaultProperties.AcknowledgeType = AcknowledgeTypes.FullReachQueue | AcknowledgeTypes.FullReceive;
  defaultProperties.Recoverable = true;
  defaultProperties.UseDeadLetterQueue = true;
  defaultProperties.UseJournalQueue = true;
  
  queue.Send(requestData);
  

• Sending the message using the code above does not appear to throw an exception - if it did my error handler in the web application would have caught and logged it, so I'm assuming it is sent.

• There are outgoing queues on W1, W2 and W3 all pointing to the private queue on P - all these are empty.
• On W1, W2 and W3 I cannot see any "dead-letter" messages.
• On P the private queue is empty so messages are being processed (which I can verify from my database).
• On P there are no "dead-letter" messages. There are journal messages but they don't seem to correspond to any recent date/times.
• All servers are running Windows Server 2012.

Most of the time messages are sent, received and processed just fine but, without any pattern visible to me, sometimes they are not. Can anyone see what is going wrong? Or explain to me how I can try and figure out what is happening?

Answer 1

Are you sure that the receiver on P does not crash/lose the message somehow? Because your queue is not transactional, if somehow processing fails then that's one lost message.

Anyway, there are many possible causes why this could fail. What kind of logging do you have (DEBUG/INFO levels)?

I think the following will help tracking down the issue:

1. When an event is generated in the web app.
2. Right before you send an event from the web app, via MSMQ.
3. In the receiver when you get a message from the queue.

This way you could at least match sent messages to received messages and to processed messages.

As a side note, when you check for dead-letter messages you do so on the source computer and on any intermediary hops, not on the destination one. If you don't have any hops, then they will be relayed to the non-transactional dead-letter queue on the web servers.