DataSource configuration to exclude deadlocks on REQUIRES_NEW methods

Question

While stress testing my JPA based DAO layer (Running 500 simultanious updates at the same time each in a separate thread). I encountered following - system always stuck unable to make any progress.

The problem was, that there were no available connections at some point for any thread, so no running thread could make any progress.

I have investigated this for a while and the root was REQUIRES_NEW annotation on add method in one of my JPA DAO's.

So the scenario was:

1. Test starts acquiring new Connection from ConnectionPool to start transaction.
2. After some initial phase, I call add on my DAO, causing it to request another Connection from ConnectionPool which there are no, because all the Connections by that time, were taken by parallel running tests.

I tried to play with DataSource configurations

1. c3p0 stucks
2. DBCP stucks
3. BoneCP stucks
4. MySQLDataSource fail some requests, with error - number of connections exceeded allowed.

Although I solved it by getting read of REQUIRES_NEW, with which all DataSources worked perfectly, still the best result seems to be of MySQLDataSource, since it did not stuck and just fail:)

So it seems you should not use REQUIRES_NEW at all, if you expect high throughput.

And my question:

Is there a configuration for either DataSources that could prevent this REQUIRES_NEW problem?

---

I played with checkout timeout in c3p0, and tests started to fail, as expected.

• 2 sec - 8 % passed
• 4 sec - 12 % passed
• 6 sec - 16 % passed
• 8 sec - 26 % passed
• 10 sec - 34 % passed
• 12 sec - 36 % passed
• 14/16/18 sec - 40 % passed

This is highly subjective ofcourse.

MySQLDataSource with plain configurations gave 20% of passed tests.

Answer 1

What about configuring a timeout for obtaining the connection? If the connection can't be obtained in say 2 seconds, the pool will abort and throw an exception.

Also note that REQUIRES is more typical. Often you'd want a call chain to share a transaction instead of starting a new transaction for each new call in a chain.

Answer 2

Probably any of the Connection pools can be configured to deal with this in any number of ways. Ultimately, all that REQUIRES_NEW is probably forcing your app to acquire more than one Connection per client, which is multiplying the stressfulness of your stress tests. If pools are hanging, it's probably because they are running out of Connections. If you set a sufficiently large pool size, you might resolve the issue. Alternatively, as Arjan suggests above, you can configure pools to "fail fast" instead of hanging indefinitely, if clients have to wait for a Connection. With c3p0, the config param for that would be checkoutTimeout.

Without more information about exactly what's going on when you say a Connection pool "stucks", this is necessarily guesswork. But under very high concurrent load, with any Connection pool, you'll either need to make lots of resources available (high maxPoolSize + numHelperThreads in c3p0), kick out excess clients (checkoutTimeout), or let clients just endure long (but finite!) wait times.