HornetQ producer threads are stuck in WAITING state

Question

We use HornetQ 2.2.5 and our problem is that in the client side (which sends messages to the Q), all the producer threads are stuck in WAITING state.

please note that the producer is on one machine and the HornetQ server is on another.

This is the thread dump in the client side:

Thread: pool-10-thread-9 : priority:5, demon:false, threadId:521, threadState:WAITING, lockName:java.util.concurrent.Semaphore$NonfairSync@60568a13

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)

java.util.concurrent.Semaphore.acquire(Semaphore.java:441) org.hornetq.core.client.impl.ClientProducerCreditsImpl.acquireCredits(ClientProducerCreditsImpl.java:74) org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:305) org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:142) org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:451) org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:199)

the connection and session to the Q are cached and reused on the client side.

On the server side there’s the following log:

[hornetq-failure-check-thread] 19:25:30,820 WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl]
Connection failure has been detected: Did not receive data from /10.2.6.11:50697. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

Is there any reason why all the producer threads are stuck?

Answer 1

That's a FAQ on hornetq's users forum...

HornetQ has different ways to deal with overflow of messages, protecting it from Out of Memory.

• Block upon overflow
• Flow to disk (as we call it.. paging)
• Dropping (messages will just disappear. only valid in cases where you can afford losing messages)
• Error (recently introduced on 2.4, 2.2 doesn't have this feature)

As you set the system to block, the client will wait upon credits that will only arrive after you consume messages.

So, either consume messages or set the address settings as paging.

Documentation about setting up paging:

http://docs.jboss.org/hornetq/2.4.0.beta1/docs/user-manual/html/paging.html#paging.main.config

Documentation about blocking mode:

http://docs.jboss.org/hornetq/2.4.0.beta1/docs/user-manual/html/paging.html#d0e5213