Java, FTP And Multiple Consumers

Question

is there any way with FTPClient to get the number of simultaneous connections that a particular ftp server will allow? And also the number of available connections at any time?

My problem is as follows I have a queue at the minute that a consumer will read data from and create a file. At the moment there are 3 consumers (Spring JMS concurrent consumers) for the queue but this number will increase and more servers listening to the queue will be added.

When I update the consumers to be 100, i get errors that the connections will fail. Can anyone recommend a design pattern or a library that will allow any number of consumers to ftp to a particular server. If for example there are 100 files and 100 consumers, if each consumer could ftp 1 file or some could ftp 10 files etc etc, that would be the desired approach. I want to split the load across multiple consumers and / or servers

Any help with this is much appreciated

Thanks Damien

Answer 1

If you can switch to SFTP, the Jsch library supports opening multiple channels over a single physical connection.

Otherwise you could implement a pool of FTP clients and block the consumer if there are no clients available. When a consumer completes its operation, return the client to the pool and wake up any waiting threads. A simple BlockingQueue is likely all you would need.

Most servers will time out idle connections, though, so you'd have to deal with a broken connection when getting one from the pool.

Finally, if you use Spring Integration instead of rolling your own, the FTP outbound channel adapter can be configured to use a CachingSessionFactory, which handles the pooling for you.