EAI/middleware: Huge messages inline or by reference

Question

Are there good practices for processing messages or requests/responses dealing with huge amounts of data (MBytes? GBytes?) via an middleware or an EAI system, assunming that the middleware transports but does not process the huge data blobs?

• Transport the data with the message in order to keep coherence but probably several times copy huge amounts of data between systems

• Keep the data in a store or data repository (file system, database,...) and transport a reference to the data, however loosing the close coupling between the message and the data

What is the common solution to this problem? How is this supported by EAI or middleware products?

Answer 1

In my experience it is good to limit the size of the messages. In the environments i know (Websphere MQ, SonicMQ) i found 5-10 MB a good value for most of the messages. In many cases it will not hurt to have very few messages with sizes up to 100 MB -> but there you will have to think about memory consumption, .... In most messaging APIs, especially JMS you will at least have the message completely in memory one time, which may cause you some headaches when starting parallel processing.

There are two things we use as a workaround:

• use the Message sequence pattern with some additions: mostly we send a trigger containing a reference of all the sequence numbers to ensure completeness
• send a "link" to the data

Normally we use the first solution whenever the messages contain structured data, the second one is used in cases of media distribution.

For the first solution we do also tend not to use provides specific solutions, but implement this logic in our custom implementation - therefor we are not bound to a specific messaging system.

Answer 2

If it is practical to make a direct connection between the source and destination system then sending only a reference through a message to start the transfer will work but, as you noted you lose the loose coupling.

According to the book Enterprise Integration Patterns you could implement the Message sequence pattern.

In my experience with EAI systems such as TIBCO. It pretty much up to you to figure our how to handle arbitrarily large amounts of data.