Configuring Spring Integration QueueChannel for RabbitMQ

Question

I am currently using spring AMQP API to connect to RabbitMQ. Basically in my consumer code I am reading the messages asynchronously and making a Bulk insert in elastic search. when I do ack = AUTO I get a moderate speed of 400-500 msg/sec (reading from Queue). when I do ack = NONE the increase in reading speed is huge i.e. it goes to 5000-6000 msg/sec.

The configuration is as follows:

• Linux machine with with 32GB RAM
• JVM arguments:

-server -Xms1g -Xmx1g -Xss384k PermSize=256m MaxPermSize=256m

Now the problem is when I am doing ack = NONE though I am getting a good speed the JVM is getting OutOfMemory after some time and I can see lot of GC is happening in this scenario.

I am planning to use the QueueChannel from Spring integration where I can limit the size of the channel as how much messages it can contain.

How can I implement this with RabbitMQ also is there any other way to achieve a good reading speed like 4000-5000 msg/sec without crashing the JVM?

Answer 1

Try using ack=AUTO, but set the prefetch and txSize to, say 1000; that way, an ack will only be sent every 1000 messages (you can tune this value as needed).

Using ackmode=NONE causes messages to accumulate in an unbounded blocking queue within the listener container and will only work if the listener can keep up (OOM means yours can't).

We have an open JIRA issue to resolve this for ack=NONE, but the above technique is usually sufficient.

You can also improve throughput using the container concurrency settings.