slow read query causes other c3p0 threads to block on acquireStatement

Question

I have a table articles with 500k rows. I only use select queries on it via a web app. All of my pages load fast (~20ms) and each consists of one or two fast queries (~10ms each). Those fast queries may either select something from the articles table or do something irrelevant, like selecting a row from another small table (5 rows).

I am having 30 pageviews per second on this webapp with no problem.

Sometimes though a specific page executes a select query which is very slow (~30 seconds) because it almost does a full table scan on the 500k rows of the articles table and just returns 3 rows. When this happens the other fast pages start slowing down heavily and at some point they completelly block.

Note that all pages use transactions and all the queries are selects so they shouldn't affect each other to much, performance wise.

I also validate the above by executing this slow query manually from a console (outside of the scope of the webapp) and the fast pages from the webapp are not affected at all. So I think that something weird is going on in the application layer.

Any idea why does this happen? Why are all threads waiting at com.mchange.v2.c3p0.stmt.GooGooStatementCache.acquireStatement?

Some data:

• MySQL: 5.5.32
• engine: InnoDB
• jdbc driver: mysql-connector-java-5.1.26-bin.jar
• pool: c3p0-0.9.1.2.jar
• orm: hibernate 3.6.8
• pool config: http://pastebin.com/3zMARM1B
• thread dump while stuck: http://pastebin.com/YGisWCzY
• tomcat: tomcat-6.0.32
• jdk: 1.6.xx

Answer 1

It is likely that the slow running prepared statement is about to be evicted from statement cache. This would cause other threads to wait until statement can be closed. Increasing statement cache size should fix the issue.