TSQL Batch insert - math doesn't work

Question

I need to insert 1.3 million of records from one table into another, and it takes really long time (over 13 min). After some research I found that it is better to do this operation in batches, so I put together something like this (actual query is more complicated, it is simplified here for briefness):

DECLARE @key INT; SET @key = 0; 
CREATE TABLE #CURRENT_KEYS(KEY INT)

WHILE 1=1
BEGIN
   -- Getting subset of keys
   INSERT INTO #CURRENT_KEYS(KEY)   
   SELECT TOP 100000 KEY FROM #ALL_KEYS WHERE KEY > @key
   IF @@ROWCOUNT = 0 BREAK

   -- Main Insert
   INSERT INTO #RESULT(KEY, VALUE)
   SELECT MAIN_TABLE.KEY, MAIN_TABLE.VALUE 
   FROM MAIN_TABLE INNER_JOIN #CURRENT_KEYS 
   ON MAIN_TABLE.KEY = #CURRENT_KEYS.KEY

   SELECT @key = MAX(KEY ) FROM #CURRENT_KEYS

   TRUNCATE TABLE #CURRENT_KEYS
END

I already have indexed list of 1.3 million keys in #ALL_KEYS table so idea here is in a loop create smaller subset of keys for the JOIN and INSERT. The above loop executes 13 times (1,300,000 records / 100,000 records in a batch). If I put a break after just one iterations - execution time is 9 seconds. I assumed total execution time would be 9*13 seconds, but it's the same 13 minutes!

Any idea why?

NOTE: Instead of temp table #CURRENT_KEYS, I tried to use CTE, but with the same result.

UPDATE Some wait stats.

I am showing for this process PAGEIOLATCH_SH and sometimes PREEMPTIVE_OS_WRITEFILEGATHER in wait stats occasionally over 500ms, but often < 100Ms. Also SP_WHO shows user as suspended for the duration of the query.

Answer 1

I'm pretty sure you're experiencing disk pressure. PREEMPTIVE_OS_WRITEFILEGATHER is an autogrowth event (database getting larger), and PAGEIOLATCH_SH means that the process is waiting for a latch on a buffer that's an IO request (probably your file growth event).

http://blog.sqlauthority.com/2011/02/19/sql-server-preemptive-and-non-preemptive-wait-type-day-19-of-28/

http://blog.sqlauthority.com/2011/02/09/sql-server-pageiolatch_dt-pageiolatch_ex-pageiolatch_kp-pageiolatch_sh-pageiolatch_up-wait-type-day-9-of-28/

What I would recommend is pre-growing both tempdb (for your temp table) and the database that's going to hold the batch insert.

http://support.microsoft.com/kb/2091024