spring-xd custom modules deployments with redis

Question

I am working on spring xd and gemfire xd where I have requirement, I collect data from one of the node(local file system) where xd-admin is runnin. I want to collect this file and send for the processing. The below is my stream definition.

source file /tmp/xd/input --ref true | processor(messageCollector) | processor((equation calc) | sink(gemfire xd)

I have 3 nodes spring xd and gemfire xd cluster. first node(xd-admin.xd-container) and 2 & 3 node xd-containers running. containers are registered with xd-admin properly. I configured single redis instance.

My requirement is, I want collect my data from file and send for the process and store to gemfire xd.

messageCollector --> collect all the messages as HashMap and send to the next processor Equation calc, here all the records will go through the equations and get calculated and results will send to the next sink(gemfire xd) insertion.

This, I have tested in single node env and working good. Now, I moved to distributed env as explained above.

How should I create modules in distributed env. Do I need to move all my modules to all the nodes available.

or

source module will be there only in where the actual file and rest of the modules goes to all the container available.

here, in equation calculator module, I am maintaining a sequence to the each and every record in side the file. and same sequence should be maintained in gemgire xd(sink) as it is.

Since now it is in single node env, it is working good as entire thing processed by single node.

Now, I moved to distributed env where I have no of containers as explained above. How to retain my sequence till insertion.

Please help me with modules deployments in distributed environment.

Thanks, -Suyodha

Answer 1

Use criteria in the deployment manifest to direct your file source to a specific node. Use count to deploy your processor(s) to multiple nodes.

However, you can't maintain sequence once you do this. If you only have 1 instance of the second processor, everything will go through it, but the sequence might not be the original sequence from the file(s).