MySQL with DBCP is creating tons of sockets, gets NoRouteToHostException

Question

We are using DBCP inside a Grails application. The database is on another server, so TCP/IP is in play here. We have monitored the database by doing a show processlist frequently, and we never see above 50 connections. However, the sockets on the client grow enormously (at one point I saw over 2700). Most of them are in TIME_WAIT status.

So eventually we get a NoRouteToHostException, because it cannot open a socket.

Note that we hit the database over 40,000 times in less than a minute in this use case.

Does anyone have suggestions as to why this might be? I would think that, since our connection pool is limited to 100 (and we only see about 50 connections open), I'd only see slightly more than 50, since occasionally one might get stale. But we're seeing thousands. Is this expected? Or any other tips about something we might be missing when looking at this situation?

Here are the dbcp settings we are using:

        properties {
            maxActive = 100
            maxIdle = 4
            minIdle = 1
            initialSize = 1
            minEvictableIdleTimeMillis = 60000
            timeBetweenEvictionRunsMillis = 60000
            maxWait = 10000
            removeAbandoned = true
            removeAbandonedTimeout = 60
            validationQuery = "/* PING */ SELECT 1"
            testOnBorrow = true
            testWhileIdle = true
            numTestsPerEvictionRun = -1
            logAbandoned = true
        }

Also note that we use autoReconnect=true on the connection string, although we are considering dropping it (we get stale connections overnight otherwise).

Thanks!

Answer 1

Ok, so I was able to sort it out. Turns out I was misunderstanding the maxIdle and how it works.

Anything returned to the pool above maxIdle is immediately released. So most of the connections were being closed and reopened, hence why the sockets were exhausted.