Hibernate c3p0 keep alive doesn't seem to work, getting 'Broken Pipe' exception

Question

I'm using Hibernate in combination with c3p0 in my web application (NO Spring). Overnight, my database connections seem to time out (or at least one of them) and i get a broken pipe exception the next day.
Below is the configuration I'm using for the project.

    <hibernate-configuration>
    <session-factory>
        <property name="hibernate.connection.url">jdbc:mysql://localhost:3306/**********?autoReconnect=true</property>
        <property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
        <property name="hibernate.dialect">org.hibernate.dialect.MySQLInnoDBDialect</property>
        <property name="hibernate.connection.username">********</property>
        <property name="hibernate.connection.password">********</property>
        <property name="hibernate.show_sql">false</property>
        <property name="hibernate.globally_quoted_identifiers">true</property>
        <property name="hibernate.current_session_context_class">thread</property>
        <property name="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>
        <property name="hibernate.connection.autocommit">false</property>
        <property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>

        <property name="hibernate.c3p0.min_size">5</property>
        <property name="hibernate.c3p0.max_size">25</property>
        <property name="hibernate.c3p0.timeout">300</property>
        <property name="hibernate.c3p0.max_statements">50</property>
        <property name="hibernate.c3p0.idle_test_period">60</property>
        <property name="hibernate.c3p0.acquire_increment">1</property>
        <property name="hibernate.c3p0.acquireRetryAttempts">5</property>
        <property name="hibernate.c3p0.acquireRetryDelay">250</property>
        <property name="hibernate.c3p0.preferredTestQuery">select 1;</property>
        <property name="hibernate.c3p0.validate">true</property>
        <property name="hibernate.c3p0.testConnectionOnCheckout">true</property>
        <property name="hibernate.c3p0.testConnectionOnCheckin">true</property>
    </session-factory>
</hibernate-configuration>

The exception I'm getting looks as following:

    The last packet successfully received from the server was 56,974,967 milliseconds ago.
    The last packet sent successfully to the server was 56,974,967 milliseconds ago. is
    longer than the server configured value of 'wait_timeout'. You should consider either
    expiring and/or testing connection validity before use in your application, increasing
    the server configured values for client timeouts, or using the Connector/J connection
    property 'autoReconnect=true' to avoid this problem.
    org.hibernate.exception.JDBCConnectionException: The last packet successfully received
    from the server was 56,974,967 milliseconds ago.  The last packet sent successfully to
    the server was 56,974,967 milliseconds ago. is longer than the server configured value
    of 'wait_timeout'. You should consider either expiring and/or testing connection
    validity before use in your application, increasing the server configured values for
    client timeouts, or using the Connector/J connection property 'autoReconnect=true' to
    avoid this problem.

The problem here is, that I am using the property autoReconnect=true, and I am testing connection validity. (See c3p0 configuration.) I'm fairly sure that the c3p0 configuration is loaded, since the following shows up in my stdout logs:

    Apr 08, 2014 10:57:13 AM com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager
    INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@f79fb235 [ 
    connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@4e20d034 [ 
    acquireIncrement -> 1, acquireRetryAttempts -> 5, acquireRetryDelay -> 250, autoCommitOnClose -> 
    false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, 
    connectionCustomizerClassName -> null, connectionTesterClassName -> 
    com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, 
    factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 
    ph9ube91hyyxjw1mgbhpo|709689fb, idleConnectionTestPeriod -> 60, initialPoolSize -> 3, 
    maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 300, 
    maxIdleTimeExcessConnections -> 0, maxPoolSize -> 25, maxStatements -> 50, maxStatementsPerConnection
     -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@a1ca3a3 [ 
    description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 
    ph9ube91hyyxjw1mgbhpo|24bbab7, jdbcUrl -> jdbc:mysql://localhost:3306/********?autoReconnect=true, 
    properties -> {user=******, password=******, autocommit=false} ], preferredTestQuery -> select 1;, 
    propertyCycle -> 0, testConnectionOnCheckin -> true, testConnectionOnCheckout -> true, 
    unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], 
    dataSourceName -> null, factoryClassLocation -> null, identityToken -> ph9ube91hyyxjw1mgbhpo|fedb05d,
     numHelperThreads -> 3 ]

The broken pipe part of the exception is this:

    Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 56,974,967 milliseconds ago.  The last packet sent successfully to the server was 56,974,967 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3851)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2471)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2683)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2144)
        at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2310)
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
        at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:122)
        ... 23 more
    Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3832)
        ... 33 more

I've tried the following, based on what I found on the internet:

• Adding autoReconnect=true to the connection URL (Please note that the connection is still broken after trying to use it, so no reconnection seems to take place)
• Adding testConnectionOnCheckout and testConnectionOnCheckin to the c3p0 config
• Adding idle_test_period and preferredTestQuery to the c3p0 config

And nothing seems to work. Has anyone got any idea on what might be causing this problem?

Answer 1

So, some scattershot ideas:

There's an error in your preferredTestQuery. It should be "SELECT 1" rather than "SELECT 1;". I'd think if this were a problem, you'd see Exceptions about it (and checkouts would all fail), but it's worth trying to fix that.

[What version of c3p0 are you using? If you want to live dangerously and are using a recent MySQL driver, try using the latest prerelease, currently c3p0-0.9.5-pre8, and don't set preferredTestQuery at all. The latest version supports JDBC4 Connection.isValid() for tests.]

Another possibility is that your application occasionally holds Connections open (checked out) for a very long time. This wouldn't quite a Connection leak, because eventually the application tries to execute a Statement against the Connection. But if your application does this, you'd see what you are seeing, as c3p0 won't test Connections while they are in client use so they'd be able to idle out. (I have no idea why autoReconnect doesn't help, except maybe you are in mid-transaction and autoReconnect doesn't work for that?) If you want to test for this scenario, you can use unreturnedConnectionTimeout (optionally with debugUnreturnedConnectionStackTraces) to forcibly close the too-long-checked-out Exception and try to diagnose the problem.

If you just want the problem to go away (and are less concerned with understanding it), you might also try setting c3p0's maxConnectionAge and/or maxIdleTime. If you set these to large values, these settings won't meaningfully impact performance. (Maybe 7200 secs, i.e. two hours, though anything more than ~1800 secs would be fine.) These settings (especially maxConnectionAge) are pretty foolproof, unless the troublesome Connections are, per above, held open and checked out of the pool for a very long time. [The only c3p0 setting that would close() a checked-out Connection out from under a client is unreturnedConnectionTimeout.]

Good luck!