HornetQ OutOfMemory on startup with big journal

StackOverflow https://stackoverflow.com/questions/1941449

  •  20-09-2019
  •  | 
  •  

Question

Using: HornetQ 2.0.0.CR2 Default configurations for the stand-alone/non-clustered server.

When I try to startup the server with a big journal (> 1Gb), I got a OutOfMemory exception:

[main] 12:59:43,505 INFO [org.hornetq.integration.bootstrap.HornetQBootstrapServer]  Starting HornetQ Server
[main] 12:59:44,526 INFO [org.hornetq.core.server.impl.HornetQServerImpl]  live server is starting..
[main] 12:59:44,532 WARNING [org.hornetq.core.server.management.impl.ManagementServiceImpl]  It has been detected that the cluster admin user and password which are used to replicate management operation from one node to the other have not been changed from the installation default. Please see the HornetQ user guide for instructions on how to do this.
[main] 12:59:44,564 WARNING [org.hornetq.core.persistence.impl.journal.JournalStorageManager]  AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal
[main] 12:59:44,565 INFO [org.hornetq.core.persistence.impl.journal.JournalStorageManager]  Using NIO Journal
Exception in thread "hornetq-expiry-reaper-thread" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.ConcurrentHashMap.values(ConcurrentHashMap.java:1011)
at org.hornetq.core.postoffice.impl.PostOfficeImpl$Reaper.run(PostOfficeImpl.java:1083)
at java.lang.Thread.run(Thread.java:637)
[main] 13:00:17,135 SEVERE [org.hornetq.integration.bootstrap.HornetQBootstrapServer]  Failed to start server
java.lang.IllegalStateException: Incompletely deployed:

DEPLOYMENTS IN ERROR:
  Deployment "JMSServerManager" is in error due to: java.lang.OutOfMemoryError: GC overhead limit exceeded

at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.internalValidate(AbstractKernelDeployer.java:278)
at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.validate(AbstractKernelDeployer.java:174)
at org.hornetq.integration.bootstrap.HornetQBootstrapServer.bootstrap(HornetQBootstrapServer.java:159)
at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:83)
at org.hornetq.integration.bootstrap.HornetQBootstrapServer.run(HornetQBootstrapServer.java:117)
at org.hornetq.integration.bootstrap.HornetQBootstrapServer.main(HornetQBootstrapServer.java:73)
Exception in thread "main" java.lang.IllegalStateException: Incompletely deployed:

DEPLOYMENTS IN ERROR:
  Deployment "JMSServerManager" is in error due to: java.lang.OutOfMemoryError: GC overhead limit exceeded

at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.internalValidate(AbstractKernelDeployer.java:278)
at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.validate(AbstractKernelDeployer.java:174)
at org.hornetq.integration.bootstrap.HornetQBootstrapServer.bootstrap(HornetQBootstrapServer.java:159)
at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:83)
at org.hornetq.integration.bootstrap.HornetQBootstrapServer.run(HornetQBootstrapServer.java:117)
at org.hornetq.integration.bootstrap.HornetQBootstrapServer.main(HornetQBootstrapServer.java:73)

This could happen in real life when a consumer stops processing the messages and I need to restart the server.

There is any workaround for this? Or which configurations should I try to modify?

Was it helpful?

Solution

It turns out it was simple to completely avoid this problem.

The Paging it's not enabled in the default configuration!

Adding these 2 lines on the hornetq-configuration.xml should do the trick:

   <address-settings>
      <!--default for catch all-->
      <address-setting match="#">
         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
         <redelivery-delay>0</redelivery-delay>
         <page-size-bytes>10485760</page-size-bytes>
         <message-counter-history-day-limit>10</message-counter-history-day-limit>

         <!-- Add these 2 lines -->
         <max-size-bytes>104857600</max-size-bytes>
         <address-full-policy>PAGE</address-full-policy>

      </address-setting>
   </address-settings>

Now it's possible to have huge queues waiting to be processed.

OTHER TIPS

From here it appears your heap is too small.

The parallel collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line.

Have you tried modifying your JVM memory options, and -Xmx (max memory allocatable) in particular ? I suspect you need to increase your JVM max memory, to give it enough headroom to process your messages.

If it is on a developer's you can simply drop your queue by deleting your HornetQ work directory (e.g. ~/work/hornetq).

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top