HornetQ throughput limited to 4000 TPS without persistence

Question

I am using HornetQ embedded in JBoss 6.1 application server. My applications (a client app, producing messages, and JBoss app consuming them) cannot handle more than 4000 TPS on a server while the CPU is still 60% idle. I tried to remove persistence to check if I was disk-bound but it does not improve the throughput.

It seems the problem is on the producer side. At least while monitoring the queue size, it stays very small, meaning consumers are not the bottleneck.

Should I use several queues to be more efficient? I already read performance tuning documentation from HornetQ, but could not find the reason for this. Or may be it is because I am using AUTO_ACKNOWLEDGE mode? I am running several threads for the producers to this should not impact a lot. The producer JVM cannot use more than 1 CPU thread anyway. I even tried to run several instances of my producer application, but it does not go faster. The network bandwidth is high (1 Gbps) and my messages are very small (< 1 KB). Also, the producer and consumer applications are running on the same server. HornetQ is configured in a JBoss cluster of 2 servers.

Answer 1

I was able to solve this by using several queues. By using JProfiler, I could see lock on my queues and all threads were waiting on these locks. I tried with 2 queues, and I could double the performance. So now I am putting in place a set of queues.

Answer 2

You could maybe try 2.3.0. I have removed a few locks on appending messages to the queue. Maybe it would scale up with a single Queue on 2.3.0.Final.

(at the time I wrote this, 2.3.0.Final was about to be released. 2.3.0.CR2 didn't have this change I'm talking about)

Answer 3

Did you increase your client's send-window-size?

<connection-factory name="ConnectionFactory">
  <producer-window-size>1000</producer-window-size>
...

This parameter limits outstanding messages per-client. It doesn't apply to a single producer thread but to all combined, so adding threads won't help if that limit was reached. Network roundtrip delays make you reach it quite soon.

see: http://docs.jboss.org/hornetq/2.2.14.Final/user-manual/en/html/flow-control.html#d0e4005