IPC design using ESB and messaging middleware

Question

First I'll write what I think the process should be to solve my problem before I lose you attention. My problem and existing setup are further below. Here's what I think should happen to allow for future flexibility. Please advise:

1. Database triggers populates SQL Service Broker and Oracle Advanced Queuing when a record is created or modified.
2. Processes watch the SQL and Oracle message queues and sends them to an ESB.
3. The ESB (UltraESB?, easy to set up and understand) in turn sends the messages to a true message queue (RabbitMQ? easy to set up and understand). The ESB might enrich, route, etc.
4. Finally, some other process consumes RabbitMQ and instructs the DMS what to: create new policy folder, update policy metadata, etc.

The triple queuing, MSSQL, Oracle and RabbitMQ, seems like overkill but on the other hand they all perform different things.

Existing setup:

I have a greenfield messaging/ESB/middleware/SOA environment and would like to set it up properly to handle, at first, the problem below.

• LOB-A: insurance application written in JAVA on a MS SQL 2008 database
• LOB-B: insurance application written in Visual Basic 6 on an Oracle 10.2 database
• DMS: Microsoft SharePoint 2010
• Both LOB apps are likley to be replaced in the next 18 months
• More insurance apps may come into the fold in future, as well as a CRM

Problem:

Both LOB apps need to interact with the document management system by signalling when new policies, customers, etc. are created as well as when they are modified. We don't have access to LOB-A source code for modification. We do have access to LOB-B source code but developers are busy on other projects. Anyway we think it is easier to have the database alert the DMS when a record has changed instead of locating all the places in the app's source code where the record might be changed and doing the alert via the application layer.

I know that Database-as-IPC is an anti-pattern, although I've read recommendations on how to best achieve this, at least for SQL Server: Best way to use a DB table as a message/job queue and http://rusanu.com/2010/03/26/using-tables-as-queues/. I'm already signalling the DMS from LOB-A by using SQL Service Broker External Activation in a point to point manner.

Whew! What do you think?

Answer 1

Disclaimer: I am the CTO of AdroitLogic, which builds the UltraESB mentioned in the question

You can easily get the ESB itself to poll the MS SQL and Oracle Databases for new actions to be performed. This could be scheduled in the ESB giving a cron schedule etc, or a simple delay (e.g. every hour). The ESB can enrich, transform and route etc, but you will need a way to track which records have been successfully processed - maybe a new column in the polled tables? Once that's available, you really do not need a persistent message queue, since the ESB can poll for un-processed records, do whatever is expected over them, and post them to the DMS - and update status as successful or failed. Unless the DMS rejects or becomes unavailable, there is no real point in-retrying, but you may want to do that, and that's possible too. If DMS accepts the record, the ESB can directly update the table columns. If you really want to use a message queue - that too is certainly possible, and depends on your selection.