Communication Between Several Storm Topologies

Question

I am trying to deploy several Storm topologies in production. I checked the documentation but couldn't find any references on whether it is possible for the topologies to communicate via native methods. Does anyone have any suggestions on how this could be implemented?

In short, I am interested to see if the it's possible for tuples to be sent across topologies.

Thanks for your help!

Answer 1

Theoretically, you could probably make it happen. Practically, no.

If you want to communicate via tuples, you should be doing that all in one topology. Topologies organize computation not code, so all the stuff that you want to compute together should go together in the topology.

That said, there are cases when you will want to split computations across topologies because of the possibly very different nature of each part of the computation. In such a case you will need to use an outside message passing service to hand off work between topologies. This is honestly way easier and more robust than trying to pass tuples directly between topologies. I have had great success using Redis.

Answer 2

Simple: if you want to keep seperated topologies, your bolts can put and retrieve values to/from an external storage, like Redis.