How does one make activemq actually reliable? When brokers disconnect messages are lost

Question

We have a JEE6 app built in Apache TomEE v1.6.0+. There are two parts, a cloud part, and a ground part. The cloud part is intended to be never restarted, since it monitors a transient source of information, but creates JMS messages and sends them to it's broker.

The ground part is intended to be restart-able during the day and is where the complex processing logic is. It too has a broker which connects to the cloud broker.

The problem we are having is that if we take down the ground instance of TomEE for more than a few mins, then start it up again, the cloud broker will not deliver all the messages that stacked up. Furthermore, it doesn't deliver any new messages either, forcing us to restart it, which makes us lose our messages.

Here are the two connection URIs... What on earth are we doing wrong??

Cloud:

<Resource
    id="ActiveMQResourceAdapter"
    type="ActiveMQResourceAdapter">
    BrokerXmlConfig = broker:(ssl://0.0.0.0:61617?needClientAuth=true&amp;transport.keepAlive=true&amp;transport.soTimeout=30000,vm://localhost,network:static:(failover:(ssl://ground.somedomain.com:61617?keepAlive=true&amp;soTimeout=30000)))?persistent=true
    ServerUrl = vm://localhost
    DataSource = jdbc/activemq
</Resource>

Ground:

<Resource
    id="ActiveMQResourceAdapter"
    type="ActiveMQResourceAdapter">
    BrokerXmlConfig = broker:(ssl://0.0.0.0:61617?needClientAuth=true&amp;transport.keepAlive=true&amp;transport.soTimeout=30000,vm://localhost,network:static:(failover:(ssl://cloud.somedomain.com:61617?keepAlive=true&amp;soTimeout=30000)))?persistent=true
    ServerUrl = vm://localhost
    DataSource = jdbc/activemq
</Resource>

Any help is much appreciated. Thank you very much!!

Answer 1

Ok we learned a couple of things.

First, we switched to using an external instance of ActiveMQ, instead of relying on the embedded one inside TomEE. You must start the broker first, before start TomEE, or TomEE will create an internal broker on startup and you'll be scratching your head going gee why aren't any messages processing. You then connect TomEE to the broker by setting BrokerXmlConfig = and ServerUrl = tcp://localhost.

Next, we switched to using the activemq http transport. This completely negates any network disconnect problems, since http is stateless. It is VERY slow however relative to tcp/ssl, but the message transport is not the slowest point in our system so it doesn't matter anyway. You MUST have the external broker listen on both http and tcp since TomEE connects via TCP and the remote broker connects via http.

These two things fixed our problems and we have a completely solid system running now. I hope this helps someone!!

Answer 2

Not sure if you are using topics or queues but the JMS spec says that only queues and durable subscribers can take advantage of the store-and-forward guaranteed delivery.

For a non-durable subscriber, a non-persistent message will be delivered “at most once” but will be missed if inactive.

Please take a look to the following URL which explains in detail how guaranteed messaging works for topics and queues in ActiveMQ:

http://www.christianposta.com/blog/?p=265