Ehcache - PermGen memory leak when using replication

Question

I'm using Ehcache as the provider for Hibernate's second level cache. Without replication everything works fine, but as soon as I add replication I get classloader memory leaks when redeploying on Tomcat 7.

At first I tried RMI replication with the following ehcache.xml:

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">

    <cacheManagerPeerProviderFactory
        class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
        properties="peerDiscovery=automatic, multicastGroupAddress=230.0.0.1, multicastGroupPort=4446"/>

    <cacheManagerPeerListenerFactory class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"/>

    <defaultCache maxElementsInMemory="10">
        <cacheEventListenerFactory class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"/>
    </defaultCache>

</ehcache>

I noticed that after a few redeployments I get PermGen out of memory error. I used VisualVM to find the root cause. Here is the path from root:

this     - value: org.apache.catalina.loader.WebappClassLoader #2
 <- <classLoader>     - class: net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory, value: org.apache.catalina.loader.WebappClassLoader #2
  <- <class>     - class: net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory, value: net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory class ConfigurableRMIClientSocketFactory
   <- csf     - class: sun.rmi.transport.tcp.TCPEndpoint, value: net.sf.ehcache.distribution.ConfigurableRMIClientSocketFactory #1
    <- key     - class: java.util.HashMap$Entry, value: sun.rmi.transport.tcp.TCPEndpoint #5
     <- [12]     - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #12867
      <- table     - class: java.util.HashMap, value: java.util.HashMap$Entry[] #1002 (16 items)
       <- localEndpoints (sticky class)     - class: sun.rmi.transport.tcp.TCPEndpoint, value: java.util.HashMap #1192

Then I decided to try JGroups instead of RMI:

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">

    <cacheManagerPeerProviderFactory class="net.sf.ehcache.distribution.jgroups.JGroupsCacheManagerPeerProviderFactory"/>

    <defaultCache maxElementsInMemory="10">
        <cacheEventListenerFactory class="net.sf.ehcache.distribution.jgroups.JGroupsCacheReplicatorFactory"/>
    </defaultCache>

</ehcache>

But ended up with the same problem!

this     - value: org.apache.catalina.loader.WebappClassLoader #2
 <- <classLoader>     - class: org.jgroups.protocols.TP$1, value: org.apache.catalina.loader.WebappClassLoader #2
  <- <class>     - class: org.jgroups.protocols.TP$1, value: org.jgroups.protocols.TP$1 class TP$1
   <- [0]     - class: java.lang.ThreadGroup[], value: org.jgroups.protocols.TP$1 #1
    <- groups     - class: java.lang.ThreadGroup, value: java.lang.ThreadGroup[] #2 (4 items)
     <- group (thread object)     - class: java.lang.Thread, value: java.lang.ThreadGroup #1

As you can see, with JGroups the cause is different, but the result is the same - OutOfMemoryError: PermGen space.

I've tried moving Ehcache to Tomcat lib directory and adding net.sf.ehcache.constructs.web.ShutdownListener to web.xml, but it didn't help.

I'm using ehcache-core 2.6.8 and ehcache-jgroupsreplication 1.7.

Answer 1

The problem with RMI seems to be caused by a bug similar to this one JDK-8025071, there is apparently no way to clear the hashmap at shutdown time so no workaround is known.

The problem with JGroups seems to be JGRP-1576. This problem is fixed by using this leak preventor. So I would say that JGroups with the leak preventor is your best bet, if shutting down the tomcat process and restarting is not an option.

Doing so would take about the same time as this redeploy, and workaround both these and other future problems that might arise in the future when libraries get upgraded in the project.