Many concurrent reads + one write cause ObjectNotFoundException due to ehcache

Question

I use Hibernate 3.6.8, ehcache 2.4.5 (also tried with latest 2.8.0), jvm 1.6.0_22 on a high traffic site, and sometimes I experience

ObjectNotFoundException: No row with the given identifier exists: [com.example.Foo#123]`

when a new Foo (in this case with id 123) is created via the simplest code possible:

Foo foo = new Foo();
session.save(foo);

The reason is that in all pages of this high traffic site I get all Foos like this:

session.createQuery("from Foo").setCacheable(true).list();

The table storing Foos contains 1000 rows, and the entity is cached in ehcache:

<class-cache class="com.example.Foo" usage="read-write" />

Other possibly relevant parts of my Hibernate configuration are:

<property name="connection.url">jdbc:mysql://localhost:3306/example?characterEncoding=UTF-8</property>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>

<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.acquire_increment">1</property>
<property name="hibernate.c3p0.idle_test_period">60</property>
<property name="hibernate.c3p0.min_size">10</property>
<property name="hibernate.c3p0.max_size">20</property>
<property name="hibernate.c3p0.max_statements">0</property>
<property name="hibernate.c3p0.timeout">0</property>
<property name="hibernate.c3p0.acquireRetryAttempts">1</property>
<property name="hibernate.c3p0.acquireRetryDelay">1</property>

<property name="hibernate.show_sql">true</property>
<property name="hibernate.use_sql_comments">true</property>

<property name="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.jdbc.use_scrollable_resultset">true</property>

<property name="hibernate.cache.provider_class">net.sf.ehcache.hibernate.SingletonEhCacheProvider</property>
<property name="net.sf.ehcache.configurationResourceName">/ehcache.xml</property>
<property name="hibernate.cache.use_query_cache">true</property>

The error happens once and then goes away. I suspect that the ehcache query cache is updated with the new entity id (123) id, but the entity cache is not yet updated with the contents of that entity. I reproduce this fairly easily locally using JMeter.

Any idea on how to solve this?

On Foo creation the ObjectNotFoundException is thrown once. If on the other hand I delete an instance of Foo then I constantly (and forever) get ObjectNotFoundException for each execution of .list(). The stacktrace can be seen at http://pastebin.com/raw.php?i=dp3HBgDB

Answer 1

To mitigate the case where an entity is deleted and then list() does not work at all I've caught ObjectNotFoundException at a higher level and when this happens I do:

session.getSessionFactory().getCache().evictCollectionRegions();
session.getSessionFactory().getCache().evictDefaultQueryRegion();
session.getSessionFactory().getCache().evictQueryRegions();  

Clearing the 2nd level cache makes the site work again. This of course doesn't prevent the problem from occuring, but it solves the downtime problem of the whole site.

Answer 2

The read-write strategy does not guarantee transactionality between the database and the cache, so I think this is what happens when the write occurs:

• the new object Foo is attached to the hibernate session of the write.

• a lazy loading proxy is inserted into the second level cache by the hibernate session associated to the write request.

• The new Foo will be inserted in the database by that same session, but the insert will take a certain time to be built, flushed and committed.

• meanwhile another request as hit the proxy to load all Foos. It finds the lazy loading proxy in the cache (see the stacktrace DefaultLoadEventListener.proxyOrLoad()), and decides to load the object (DefaultLoadEventListener.load()).

• This triggers a Hibernate load() of a Foo not yet inserted in the database by the write thread.

• No Foo with that Id is found on the database, so a ObjectNotFoundException is thrown.

To confirm this, put an exception breakpoint on your IDE, to see that at the moment the exception is thrown the object had not yet been inserted in the DB. One way to solve it would be to use the transactional strategy.

Answer 3

Taken from the documentation on Cache Configuration:

The following attributes and elements are optional.

timeToIdleSeconds:
Sets the time to idle for an element before it expires.
i.e. The maximum amount of time between accesses before an element expires
Is only used if the element is not eternal.
Optional attribute. A value of 0 means that an Element can idle for infinity.
The default value is 0.

timeToLiveSeconds:
Sets the time to live for an element before it expires.
i.e. The maximum time between creation time and when an element expires.
Is only used if the element is not eternal.
Optional attribute. A value of 0 means that and Element can live for infinity.
The default value is 0.

Or you can also go with the alternate options:

Data Freshness and Expiration