Message Queueing Scenario - Queueing and remote web service

Question

(Please see Slide Links below) I am trying to create and audit-able request and response service that interacts with a remote web service. I am having some trouble deciding on the correct implementation for my approach. Basically, how I need the implementation to work is as follows.

• A Requester application (A) will generate a request XML and add it to a request 'queue'. A request processor will then get an unprocessed record from the 'queue' and POST it to a remote web service (X) with a unique ID, if the request is not successful the request is kept/(requestComplete flag = 0) in the request 'queue' to be retried at a later time.

• If the request is successful it is kept/(requestComplete flag = 1) and is not retried

• At A later date a Receiver web service (B) receives a response from the 'X' service that was called in the request.

• 'B' then searches through the request records to find the original request and correlates the 'A' request and the 'X' response (matching using the unique ID).

• Some additional processing is done with the response and the record from the 'queue' is updated as completed.

In this way there is a complete audit trail from request to response. I can see when the original request was made, if there is an error with the request by looking at the 'queue' record. Likewise with the response I can see when it was received if at all.

There are two ways I have thought to implement this.

1. Scenario 1 With a database table that acts as the request queue, the response queue as well as the audit trail in one. One row in a table with a GUID that can be referenced throughout any stage of the process (request->processing->receipt) and updated along the way. The trouble is that this implementation from what I gather does not act like a true queue like MSMQ (push/pop), transactable and not as scalable.
2. Scenario 2 For actual queue implementation I did some research and thought to use MSMQ that will perhaps have multiple queues; a processor queue to deal with the processing and sending off of the request, then push that completed request to a Receiver Queue to wait for the response from 'X'. The only trouble with this method there is no clear audit trail, i.e. the request is removed from the queue once it is received, likewise with the response. Unless I use a database table to store the request and response queues for auditing purposes. I have read that there is journal type transactions with MSMQ which does keep what records were queued but I am looking for a more complete solution or indeed advice on the matter.

Just a few notes:

• The request 'A' sends to 'X' with a unique Id, 'X' sends a response to 'B' referencing that unique Id. This allows 'B' to track down request record 'A'.
• I need to be able to retry failed attempts to 'X' (any error 400 or 404's need to be retried)
• I need to be able to keep an audit trail for the requests/responses.
• I am using C#, WCF, MSMQ, SQL Server 2008 R2, VS 2012.

If anyone has any suggestions or guidance on which avenue to pursue, any comments or any knowledge on best practices in dealing with scenarios above, it would be greatly beneficial.

Answer 1

I would suggest looking at a service bus with sagas (my preference is Rhino Service Bus, but NServiceBus also has a lot of traction, and there's also Mass Transit)

Basically a service bus will handle the plumbing of enqueuing (aka sending) messages, and dequeueing (aka receiving & processing them). A saga will then assist with maintaining state for a conversation (where a conversation consists of multiple messages).

The issues of retrying later can be handled quite elegantly with delayed messages in Rhino Service Bus, and timeouts in NServiceBus (I have no experience of Mass Transit).

I would store the result logs in a database to make it easier to query & report on. I would also rather use a servicebus with MSMQ (or any other queue) than use a database table as a "queue" - the former is designed precisely for your scenario, whereas the latter is a much more generic product to handle many different scenarios and thus it won't be as efficient as a queue implementation (eg MSMQ) (although it can do it - but scaling becomes harder).