spring integration using multiple executors

Question

I have a spring integration configuration where I am polling files from a directory and then processing those files in later steps.

I am using a dispatcher channel for processing steps which allows the poller thread to be released and return back to continue to do polling while processing steps are happening.

This works fine until the thread pool limit is reached, at that point because of CALLER_RUNS rejection policy when the poller submits the file to processing step it ends up doing the processing in the same thread as the poller which stops the polling process to be put on halt until this step is completed.

How can I c*ompletely decouple the two stages* of this process?

My thinking was to use two task-executors, one for polling and another for dispatcher but that doesn't seems to help.

<!-- Poll files from landing zone directory -->
<int-file:inbound-channel-adapter id="files" directory="${lz.dir.${ft}}" filename-regex=".*\.txt$">
    <int:poller fixed-delay="3000" max-messages-per-poll="2" task-executor="pollingExecutor" />
</int-file:inbound-channel-adapter>

<int:bridge input-channel="files" output-channel="sourceFiles" />

<!-- Dispatch retrieved files -->
<int:channel id="sourceFiles">
    <int:dispatcher task-executor="processingExecutor" />
</int:channel>

Basically I want the polling process to never stop. ANy help would be appreciated.

Thanks

Answer 1

Use a QueueChannel; the poller dumps the file into the queue an you have a poller on the other side pulling messages out of the queue.

The queue is unbounded by default.

Or, of course, you can use an unbounded queue in your executor.

But if your consumer(s) can't keep up, you'll eventually get into trouble.