Scheduler stopped when deadlock happened in Spring Transaction

Question

I've developed a monitoring application that uses Spring-3.0 Hibernate-3.6.5 with SpringHibernateTemplate as DAO. Database SQL Server 2008. This app has long running transaction every day.

@Repository("areaDaoHibernate")
public class AreaDAO implements IArea {
       protected HibernateTemplate template = null;

       @Autowired @Required
       public void setSessionFactory(SessionFactory sessionFactory) {
              template = new HibernateTemplate(sessionFactory);
       }

       // basic dao method
}

This app has some scheduling (implements Runable) via springbeans.

@Component("dailyTask")
@Scope("prototype")
public class DailyTask implements Runnable {

@Override
       public void run() {
              // running task
       }
}

This app uses Apache DBCP for pooling.

   <bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
              <property name="dataSource">
                     <ref bean="CustodyDataSource" />
              </property>
…
       </bean>

<bean id = "CustodyDataSource"
      class             = "org.apache.commons.dbcp.BasicDataSource"
      p:driverClassName = "${jdbc1.driverClassName}"
      p:url             = "${jdbc1.url}"
      p:username        = "${jdbc1.username}"
      p:password        = "${jdbc1.password}"
      p:maxIdle         = "${jdbc1.maxIdle}"
      p:maxActive       = "${jdbc1.maxActive}" 
      p:validationQuery = "${jdbc1.validationQuery}" 
      p:testWhileIdle   = "${jdbc1.testWhileIdle}" />
<bean id = "transactionManager"
      class = "org.springframework.orm.hibernate3.HibernateTransactionManager"
      p:sessionFactory-ref = "sessionFactory" />

I have problem when this app deadlock, the scheduling stop to run.
The error shows:

2012-12-27 12:57:20,861  WARN (JDBCExceptionReporter:233) - SQL Error: 1205, SQLState: 40001
2012-12-27 12:57:20,862 ERROR (JDBCExceptionReporter:234) - Transaction (Process ID 55) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
2012-12-27 12:57:20,866 ERROR (DailyTask:238) - org.springframework.dao.CannotAcquireLockException: could not update: [com.btpn.custody.entity.MasterData#00176ed4-b7a2-4da5-a266-ffaab3654050]; SQL [update T_CIF set __UPDATED_DATE=?, STATUS_MD=? where __SID=?]; nested exception is org.hibernate.exception.LockAcquisitionException: could not update: [com.btpn.custody.entity.MasterData#00176ed4-b7a2-4da5-a266-ffaab3654050]
            at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:633)
            at org.springframework.orm.hibernate3.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:793)
            at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:664)
            at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
            at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
            at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
            at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
            at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
            at $Proxy33.relateWithBranch(Unknown Source)
            at com.btpn.custody.bean.DailyTask.relateMasterDataWithBranch(DailyTask.java:1166)
            at com.btpn.custody.bean.DailyTask.manualInitial(DailyTask.java:213)
            at com.btpn.custody.bean.DailyTask.run(DailyTask.java:197)
            at java.lang.Thread.run(Unknown Source)
Caused by: org.hibernate.exception.LockAcquisitionException: could not update: [com.btpn.custody.entity.MasterData#00176ed4-b7a2-4da5-a266-ffaab3654050]
            at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:107)
            at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
            at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2612)
            at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2494)
            at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2821)
            at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113)
            at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273)
            at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265)
            at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)
            at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
            at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
            at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
            at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:383)
            at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:133)
            at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656)
            ... 11 more
Caused by: java.sql.SQLException: Transaction (Process ID 55) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
            at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
            at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
            at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
            at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
            at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
            at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
            at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:504)
            at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:102)
            at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46)
            at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2590)
            ... 23 more

So my questions:

1. Should I change connection pool to avoid deadlock problem?
2. Why my scheduling bean stop run after deadlock happen?

Answer 1

The deadlock occurred in SQL Server. This is neither Spring, Hibernate, the connection pool nor SQL Servers fault. The problem is that your application is locking rows that either your application, or other applications are also trying to lock (in a deadlock manner).

The easiest fix is probably to have a retry-strategy for your job (just retry the whole transaction if it fails). You could also try to figure out exactly which applications / users were involved in the deadlock by looking at the SQL Server logs.