C3P0 Deadlock after server sits idle over night

Question

I have a java servlet application that runs on tomcat with c3p0 configured in tomcat. I have an application that is used internally by just a few people at my organization. Typically there are 3 or less people using it and sometimes up to 5. So, not a lot of people. Often, and unpredictably, the application fails with the below error. Usually it occurs after a long period of idle time (overnight) and fails on the first request the next day. I have other production applications with a similar stack that don't have this issue but they are rarely idle. Here's the error (and yes, I've searched stackoverflow and seen others with similar issues but none of the solutions seem to work for me):

WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@2a3252ed -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Mar 19, 2014 6:42:11 AM com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector run
WARNING: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@2a3252ed -- APPARENT DEADLOCK!!! Complete Status:
        Managed Threads: 3
        Active Threads: 3
        Active Tasks:
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@b7eed31
                        on thread: C3P0PooledConnectionPoolManager[identityToken->2ykl7d901mnvek411mheiv|3a7f5639]-HelperThread-#2
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3a0631dc
                        on thread: C3P0PooledConnectionPoolManager[identityToken->2ykl7d901mnvek411mheiv|3a7f5639]-HelperThread-#1
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@6cb38046
                        on thread: C3P0PooledConnectionPoolManager[identityToken->2ykl7d901mnvek411mheiv|3a7f5639]-HelperThread-#0
        Pending Tasks:
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7caaab6a
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@28dbfc17
                com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2835ef6
                com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@7ecf69c3
                com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6328c23f
                com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@8e8e8f5
                com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@73163164
                com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@3604d743
                com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@77d08b69
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1c70d7f3
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3471a11b
Pool thread stack traces:
        Thread[C3P0PooledConnectionPoolManager[identityToken->2ykl7d901mnvek411mheiv|3a7f5639]-HelperThread-#2,5,main]
                java.net.SocketInputStream.socketRead0(Native Method)
                java.net.SocketInputStream.read(SocketInputStream.java:146)
                com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
                com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
                com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
                com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3116)
                com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3573)
                com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3562)
                com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4113)
                com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
                com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
                com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2812)
                com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2761)
                com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1612)
                com.mysql.jdbc.ConnectionImpl.getTransactionIsolation(ConnectionImpl.java:3352)
                com.mchange.v2.c3p0.impl.NewPooledConnection.<init>(NewPooledConnection.java:125)
                com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:211)
                com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
                com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
        Thread[C3P0PooledConnectionPoolManager[identityToken->2ykl7d901mnvek411mheiv|3a7f5639]-HelperThread-#1,5,main]
                com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:672)
        Thread[C3P0PooledConnectionPoolManager[identityToken->2ykl7d901mnvek411mheiv|3a7f5639]-HelperThread-#0,5,main]
                com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:672)

Here's my config in tomcat. Items in "capslock" are changed to protect the innocent:

<Resource name="jdbc/NAME_HERE" auth="Container"
              description="DESCRIPTION"
              jdbcUrl="jdbc:mysql://PATH_TO_RDS?autoReconnect=true"
              user="USER"
              password="PASS"
              type="com.mchange.v2.c3p0.ComboPooledDataSource"
              factory="org.apache.naming.factory.BeanFactory"
              driverClass="com.mysql.jdbc.Driver"
              maxPoolSize="50"
              minPoolSize="10"
              acquireIncrement="5"
              acquireRetryAttempts="0"
              acquireRetryDelay="3000"
              breakAfterAcquireFailure="false"
              maxConnectionAge="60"
              maxIdleTime="30"
              maxIdleTimeExcessConnections="10"
              idleConnectionTestPeriod="15"
              testConnectionOnCheckout="true"
              preferredTestQuery="SELECT 1"
              debugUnreturnedConnectionStackTraces="true"
              autoCommitOnClose="true"
    />

I am using C3P0 0.9.2.1, mchange 0.2.3.4, and mysql connector 5.1.26 at the server. I'm also using hibernate 3.2.5.ga in the application.

I don't know if it matters but we use RDS from Amazon, so it's not on localhost.

Any help would be appreciated. This has been going on for a while and I've tried everything I can find on various forums with no luck.

Answer 1

Just to provide an answer, this all turned out to be a JVM memory issue. The JVM would run out of memory overnight. I don't know why but this server had some special garbage collector options turned on for tomcat and when I remove those, not only did the problem go away but the memory footprint on that machine is much smaller. The default garbage collector must be more aggressive then the one that was being used. The problem hasn't occurred in several weeks and was never related to c3p0, I don't think. C3P0 was just the one catching and reporting problems.