Inserting multiple records from my Web site fails

Question

I am querying a Web site for 5 yrs worth of data from 150 data points, which is about 40000 records each(total 6m records), and inserting the result into my database. This is part of a Web application. I am using VS 2010 and SQL server 2008 Express. I have two questions unrelated to each other regarding this:

1) When I use the VS 2010 debugger and run the application and try this task using ADO.NET it takes roughly 15 minutes for 40000 records. Is there a way to speed this up. I am using the normal insert process as follows.

SqlConnection myConnection = new SqlConnection("user id=sa; password=++++++; server=.\\SQLEXPRESS;database=+++++++++;");
myConnection.Open();
{
    string CommandText = "Insert into myTable(TimeOfDay,Price,InstrumentID)"+" "
                        +"values('"+tod+"',"+price+","+insID+")";
    SqlCommand myCommand = new SqlCommand(CommandText,myConnection);
    myCommand.ExecuteNonQuery();
}

2) My second question is, when I run the application using IIS instead of using the VS 2010 attached debugger, the insertion process stops after just inserting 100 records? Is this restriction due to a setting in IIS 6.0? Or in SQL server?

Kindly help.

Answer 1

You should use a parameterized query. It is not only for more secure operations, but also for better performance, because the Sql Optimizer can reuse the same query when you use parameters

You could try this code (just need to test a bit, because I have written here on the fly)

// A parameterized command text
string CommandText = "Insert into myTable(TimeOfDay,Price,InstrumentID) values(@tod, @price, @id)";
using(SqlConnection myConnection = new SqlConnection(......))
{
     myConnection.Open();
     using(SqlCommand myCommand = new SqlCommand(CommandText,myConnection))
     {
         // Build the parameters before entering the loop. They are always the same
         // just the value changes, but setting the size, precision and scale allows
         // the SQL Optimizer to reuse this command
         Parameter tod = myCommand.Parameters.Add("@tod", SqlDbType.NVarChar);
         tod.Size = 50;  // TimeOfDay is a string? Set its size...

         // Price is a numeric with Precision and Scale? Set this properties
         Parameter price = myCommand.Parameters.Add("@price", SqlDbType.Decimal);
         price.Precision = 10;
         price.Scale = 2;

         Parameter id = myCommand.Parameters.Add("@price", SqlDbType.Int);

         for( ........ loop statement on your data .....) 
         {
            ... extract the parameters values ...
            myCommand.Parameters["@tod"] = tod;
            myCommand.Parameters["@price"] = price;
            myCommand.Parameters["@id"] = insID;
            myCommand.ExecuteNonQuery();
         }
    }
}

A very useful reading How Data Access Code Affects Database Performance