HornetQ timed out rollback leaves messages in DeliveringCount

Question

We have one app that does some very dynamic routing of hornetq messages so it reads from one queue and writes to another - within a single transacted session. The idea is that if we cannot move to the destination queue we won't remove it from the original queue. It works fine until we do some testing with network latency - things start to fall apart.

Because of the latency the transacted cannot be committed and cannot be rolled back. We see a couple of thousands messages stuck in deliverycount and never clear out of there even after 20 minutes. We have the call timeout for the connection factory left at default - which is 30 seconds. The app has about 10 instances serving 8 different destination queues so you would think that if the delivering was being done sequentially we would not see more than 80 files stuck in deliveringcount but we do see way more than that.

These files get out of the deliveringCount only when the app shuts of or if we kill and restart session.

It seems that HornetQ is not able to timeout the transaction (you would think call timeout would do the trick maybe) and remove those files from deliveringcount.

Is there any property that we should look at? I was looking for a transaction timeout but it comes only for XA transactions which we do not use.

By the way we use a messagelistener to receive the messages.

Here is the error we get on the rollback timeout (there is a similar timeout error on the commit as well):

javax.jms.JMSException: HQ119014: Timed out waiting for response when sending packet 68
    at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:379)
    at org.hornetq.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:726)
    at org.hornetq.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:712)
    at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:617)
    at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:597)
    at org.hornetq.core.client.impl.DelegatingSession.rollback(DelegatingSession.java:479)
    at org.hornetq.jms.client.HornetQSession.rollback(HornetQSession.java:250)
    at comms.HQRouter.MessageMover$MessageListenerImpl.onMessage(MessageMover.java:185)
    at org.hornetq.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:98)
    at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1085)
    at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:57)
    at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1220)
    at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: HornetQException[errorType=CONNECTION_TIMEDOUT message=HQ119014: Timed out waiting for response when sending packet 68]
    ... 16 more

Answer 1

HornetQ have had a few bug fixes after 2.3.0. you should try its latest releases.

I believe this issue would be fixed on later releases but if it's not the case you should provide a test replicating this issue and raise a forum post or if you are absolutely sure you have replicated the issue on a test with the latest release you should open a bug fix request at JBoss JIRA on the project HornetQ.