Application level pooling (c3p0) and HAProxy for load balancing read replica db requests

Question

In order to increase performance of my Java/Spring web app deployed in Tomcat 7, I am trying to use MySQL read replicas for queries that I've annotated as read-only okay. I have been using c3p0 for connection pooling prior to setting up an haxproxy server that will load balance requests to the read replicas and it works fine. I know that these two things kind of work against each other because each are trying to pool connections, however if I don't use connection pooling at the application layer, the high volume of queries opening new connections results in a very visible performance hit using Spring's DriverManagerDataSource instead of c3p0's ComboPooledDataSource.

The HAProxy setup works, however the timeout values set (at 50s) close connections that c3p0 thinks is open, so I have set the maxIdleTime on c3p0 at 40s as opposed to haproxy's 50 to avoid haproxy closing connection before c3p0 can. This actually worked pretty well, but occasionally I still see an exception indicating that haproxy killed the connection before c3p0.

20:44:14,470 ERROR http-bio-8080-exec-1 transaction.JDBCTransaction:95 - JDBC begin   failed
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 55,267 milliseconds ago.  The     last packet sent successfully to the server was 1 milliseconds ago.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

I can see some debug messages where idle connections are getting cleaned up, but I'm wondering why c3p0 is seemingly failing to clean up some of the connections in the pool that have gone beyond 40s.

My configs:

c3p0

<bean name="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
    <property name="driverClass" value="${db.driver.classname}"/>
    <property name="user" value="${db.username}"/>
    <property name="password" value="${db.password}"/>
    <property name="jdbcUrl" value="${db.url}"/>
    <property name="minPoolSize" value="5"/>
    <property name="maxPoolSize" value="15"/>
    <property name="acquireIncrement" value="5"/>
    <property name="maxIdleTime" value="40"/>
</bean>

haproxy

global
    log 127.0.0.1   local0
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode    http
    option  dontlognull
    retries 3
    option redispatch
    maxconn 2000
    contimeout  5000
    clitimeout  50000
    srvtimeout  50000

listen  MySQL 0.0.0.0:3306
  mode tcp
  balance roundrobin
  server rr1 some.nice.com:3306 maxconn 620
  server rr2 some.cool.com:3306 maxconn 620

Thanks in advance!

Answer 1

The specific problem I had here was solved by increasing the difference between clitimeout/srvtimeout in haproxy and the maxIdleTime in c3p0 as well as setting a idleConnectionTestPeriod because despite a maxIdleTime being set in c3p0, it will not automatically check for idle connections unless an interval is specified.

Answer 2

Have you tried instructing c3p0 to do health checks for the connections? I had a similar problem where the connection was closed but c3p0 thought it was still active so I was trying to use a non existing connection (I am not using Spring thought). Adding this parameters did the trick for me:

hibernate.c3p0.idle_test_period=3000
hibernate.c3p0.timeout=3500
hibernate.c3p0.validate=true
hibernate.c3p0.preferredTestQuery=SELECT 1
hibernate.c3p0.maxConnectionAge=3600
hibernate.c3p0.testConnectionOnCheckin=true
hibernate.c3p0.testConnectionOnCheckout=true
hibernate.c3p0.acquireRetryDelay=1000
hibernate.c3p0.acquireRetryAttempts=30
hibernate.c3p0.acquire_increment=1
hibernate.c3p0.breakAfterAcquireFailure=false