How to handle multiple SQL transactions through different .net(c#) threads

Question

I have the following method for bulk insert of the data in tables. First my code populates the data in data tables and inserts this data in corresponding tables using the SqlBulkCopy claas of the .net .

I have requirement that data should get inserted in all tables or neither of them. For this I have used SqlTransaction class of the .net.

Scenario is, multiple threads execute the following code block at the same time.

 public void Import()
        {
            using (SqlConnection sqlConnection = new SqlConnection(connectionString))
            {

                SqlTransaction sqlTrans =null;
                try
                {
                    sqlConnection.Open();
                    sqlTrans = sqlConnection.BeginTransaction(IsolationLevel.Serializable)


                    SqlCommand cmd = sqlConnection.CreateCommand();
                    cmd.CommandText = "select top 1 null from lockTable with(xlock)";
                    cmd.CommandTimeout = 3600*3;
                    cmd.Transaction = sqlTrans;
                    SqlDataReader reader = cmd.ExecuteReader();


                    foreach (DataTable dt in DataTables)
                    {
                        ImportIntoDatabase(sqlConnection, dt, sqlTrans);
                    }

                    reader.Close();
                    sqlTrans.Commit();                    
                }
                catch (Exception ex)
                {
                    sqlTrans.Rollback();
                    throw ex;
                }
            }
        }

       private void ImportIntoDatabase(SqlConnection sqlConn, DataTable dt, SqlTransaction sqlTrans)
        {
            using (SqlBulkCopy bulkCopy = new SqlBulkCopy(sqlConn, SqlBulkCopyOptions.Default, sqlTrans))
            {
                bulkCopy.BulkCopyTimeout = dt.Rows.Count * 10;
                try
                {
                    bulkCopy.DestinationTableName = dt.TableName;                    
                    bulkCopy.WriteToServer(dt);
                }
                catch (Exception ex)
                {
                   throw ex;
                }
            }
        }

To handle this concurrency, I have created one dummy table(table named 'lockTable'), in the database where the other table resides(the bulk insert tables). I am getting exclusive lock on this dummy table in the SqlTransaction having command time out as high as 3 hours.

Problem: I am getting following exception

: Cannot access destination table 'Tbl1' (tbl1 is the table for bulk inserting)

followed by another exception, while rolling back the transaction in catch block

: Error While executing activity The server failed to resume the transaction. Desc:3a00000001. The transaction active in this session has been committed or aborted by another session.

Can any one help me for this weird behavior of the code. I have already searched a lot on this issue on the internet, but I have not found anything helpful for me.

Answer 1

I got my problem solved out.

Following are the changes which I have made to my Import method

public void Import()
        {
            using (SqlConnection sqlConnection = new SqlConnection(connectionString))
            {
                sqlConnection.Open();
                using (SqlTransaction sqlTrans = sqlConnection.BeginTransaction())
                {
                    try
                    {                       
                        SqlCommand cmd = sqlConnection.CreateCommand();
                        cmd.CommandText = "select top 1 null from lockTable with(xlock)";
                        cmd.CommandTimeout = LOCK_TIME_OUT;
                        cmd.Transaction = sqlTrans;
                        SqlDataReader reader = cmd.ExecuteReader();


                        foreach (DataTable dt in DataTables)
                        {
                            ImportIntoDatabase(sqlConnection, dt, sqlTrans);                            
                        }

                        reader.Close();
                        sqlTrans.Commit();                        
                    }
                    catch (Exception ex)
                    {
                        sqlTrans.Rollback();
                        throw ex;
                    }
                }
                sqlConnection.Close();
            }
        }

Answer 2

In Import (DataTable dt in DataTables) is not going to be thread safe.

sqlConnection already has an active reader from Import so that connection cannot be used in ImportIntoDatabase.

Echo smp - if you are locking a table then why multi threads?

If you want to build up the input while the SQL inserts are taking place then use Asynch method such as SqlCommand.BeginExecuteReader. You get asynch without the overhead of a thread. And DataTables are relatively slow. I insert using TVP and light weight objects. A huge factor in insert performance is index fragmentation. If at all possible insert order by the order of the clustered index. The loop is simple build input, wait for asynch, run asych. Or build input may be read input from a queue. SQL insert to the same table(s) are typically not going to go faster in parallel. My experience is ordered serial inserts with no gap in time between inserts.

Answer 3

If multiple threads have access to the "Import" Method, then shouldn't you be locking the content of this method?

I don't think you need a dummy table, you just need to lock the two methods above.

I would also mention that you should join all threads, so that you can tell when they have finished.