SQL Server deadlock under NHibernate

Question

I have a multithread application, that works with large database (file size >1 Gb, database has 38 tables, more than 500 K of entities per table). It uses Castle 3.1.0.0, NHibernate 3.3.1.4000, FluentNibernate 1.3.0.733, SQL Server 2012.

NHibernate configured in next way:

config.SetProperty(Environment.CommandTimeout, "300");
config.SetProperty(Environment.BatchSize, "0"); 
config.SetProperty(Environment.GenerateStatistics, "true");
config.SetProperty(Environment.ReleaseConnections, "auto");
config.SetProperty(Environment.UseQueryCache, "true");
config.SetProperty(Environment.SqlExceptionConverter, typeof(MsSqlExceptionConverter).AssemblyQualifiedName);
//...
.MaxFetchDepth(1)

I use one Session per thread (Castle.Windsor) and short transactions. Each database update, save, delete procedure locked by code:

public abstract class BaseEntityRepository<T, TId> : IBaseEntityRepository<T, TId> where T : BaseEntity<TId> {
    protected static readonly object Locker = new object();

    public bool Save(T item) {
        bool result = false;

        if ((item != null) && (item.IsTransient())) {
            lock (Locker) {
                using (ITransaction tr = Session.BeginTransaction()) {
                    try {                       
                        Session.Save(item);
                        if ((tr.IsActive) && (!tr.WasCommitted) && (!tr.WasRolledBack))
                            tr.Commit();  
                        result = true;
                    } catch {
                        if ((tr.IsActive) && (!tr.WasCommitted) && (!tr.WasRolledBack))
                            tr.Rollback();
                        Session.Clear();
                        throw;
                    }
                }
            }
        }
        return result;
    }

    //same for delete and update 

    public T Get(TId itemId) {
        T result = default(T);

        try {
            result = Session.Get<T>(itemId);
        } catch {
            throw;
        }
        return result;
    }

    public IList<T> Find(Expression<Func<T, bool>> predicate) {
        IList<T> result = new List<T>();
        try {
            result = Session.Query<T>().Where(predicate).ToList();
        } catch {
            throw;
        }
        return result;
    }
}

All worked fine before I merged 3 databases (300-400 Mb each) into one big database (described upper). I merged databases by Microsoft SQL Server Management Studio Export/Import data wizards. After merge, I set up primary keys, and set up identity specification for Id columns. After this I got many SQL Server errors (this errors not reproduced, they appears time to time):

Transaction (Process ID X) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

So I read this article, this. I did all steps, and found, that two threads do same pagelock on same associatedObjectId. I even don't understand how it can be, because I have programatically lock on every entity update, so it even can't be such issue. That's why I changed database to snapshot isolation:

ALTER DATABASE MY_DB SET ALLOW_SNAPSHOT_ISOLATION ON

ALTER DATABASE MY_DB SET READ_COMMITTED_SNAPSHOT ON

also I changed NHibernate IsolationLevel from ReadCommitted to Snapshot to solve this problem. After this I got this errors:

Could not execute command: UPDATE [Foo] SET Field1 = @p0, Field2= @p1 WHERE Id = @p2
System.Data.SqlClient.SqlException (0x80131904): Snapshot isolation transaction aborted due to update conflict. You cannot use snapshot isolation to access table 'dbo.Bar' directly or indirectly in database 'My_DB' to update, delete, or insert the row that has been modified or deleted by another transaction. Retry the transaction or change the isolation level for the update/delete statement.

in System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
in System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
in System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
in System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
in System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
in System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
in System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
in System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
in NHibernate.AdoNet.AbstractBatcher.ExecuteNonQuery(IDbCommand cmd)

Could not synchronize database state with session NHibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [Foo#544353]

in NHibernate.Persister.Entity.AbstractEntityPersister.Update(Object id, Object[] fields, Object[] oldFields, Object rowId, Boolean[] includeProperty, Int32 j, Object oldVersion, Object obj, SqlCommandInfo sql, ISessionImplementor session)
in NHibernate.Persister.Entity.AbstractEntityPersister.UpdateOrInsert(Object id, Object[] fields, Object[] oldFields, Object rowId, Boolean[] includeProperty, Int32 j, Object oldVersion, Object obj, SqlCommandInfo sql, ISessionImplementor session)
in NHibernate.Persister.Entity.AbstractEntityPersister.Update(Object id, Object[] fields, Int32[] dirtyFields, Boolean hasDirtyCollection, Object[] oldFields, Object oldVersion, Object obj, Object rowId, ISessionImplementor session)
in NHibernate.Action.EntityUpdateAction.Execute()
in NHibernate.Engine.ActionQueue.Execute(IExecutable executable)
in NHibernate.Engine.ActionQueue.ExecuteActions(IList list)
in NHibernate.Engine.ActionQueue.ExecuteActions()
in NHibernate.Event.Default.AbstractFlushingEventListener.PerformExecutions(IEventSource session)

NHibernate.TransactionException: Transaction not connected, or was disconnected

in NHibernate.Transaction.AdoTransaction.CheckNotZombied()
in NHibernate.Transaction.AdoTransaction.Rollback()
in My.DBLayer.Data.Repositories.BaseEntityRepository`2.Update(T item)

I don't have Version property for DB entities, and there is no OptimisticLock.Version() line in my code, so I use PessimisticLock implicity. I can add Version and OptimisticLock, but don't think that this will solve problem.

I tried to do simple test:

Thread t1 = new Thread(m1);
Thread t2 = new Thread(m2);
t1.Start();
t2.Start();

private static void m1() {
    FooRepository rep1 = BootstrapContainer.Instance.Resolve<FooRepository>();
    Foo foo1 = rep1.Get(1);
    foo1.Field1 = "bbbb";
    Thread.Sleep(60*1000);
    rep1.Update(foo1);
}

private static void m2() {
    FooRepository rep2 = BootstrapContainer.Instance.Resolve<FooRepository>();
    Thread.Sleep(5*1000);
    Foo foo2 = rep2.Get(1);
    foo2.Field2 = "aaaaa";
    Thread.Sleep(5*1000);
    rep2.Update(foo2);
}

all works fine without any errors.

Why I have all this errors (I not changed code, just merged databases into one, and before merging all worked fine)? Why this errors can even be if I use locking to prevent update entities in different threads simultaneously.

Answer 1

You can get deadlocks from query parallelization... that is, a single query can deadlock against itself when it splits the work into several parallel operations. I have had this happen to me on several occasions. If you are using hql, you can add OPTION(MAXDOP 1) to your query/statement to see if that fixes the problem.

Alternatively you can set the entire server to MAXDOP 1... meaning you'll never get parallelism (https://msdn.microsoft.com/en-us/library/ms189094.aspx). I wouldn't recommend this without deep analysis of the workload, but it can help you see if the parallelism is the source of the deadlocks.