TSQL Insert Transaction Log Filling up

Question

Need to generate some test data. This insert is 800,000 X 1,000. I know a lot but this is a real application where the random will be a calculated number.

How can I break this up so the transaction log does not fill up?

insert into sIDcrossMatch
  select 
  docSVsys1.sID, docSVsys2.sID, Abs(Checksum(NewId())) % 100 As RandomInteger 
  from docSVsys as docSVsys1 
  join docSVsys as docSVsys2
  on docSVsys1.sID <> docSVsys2.sID 
  where docSVsys1.sID < 1000
  order by docSVsys1.sID, docSVsys2.sID

It will insert one docSVsys1.sID without filling up the transaction log.

Answer 1

Since that is your test database, make sure your Recovery model to Simple first and then let log grow as much as you could provide space to it (add more files if needed). And be sure that You understand consequences of these settings.

Next step, or first step if you can't set recovery model and allow log to grow, split your insert statement into multiple insert statements by adding where clause like this:

1. insert #1: where docSVsys1.sID % 2 = 0 /* % means moduo */
2. insert #2: where docSVsys1.sID % 2 = 1 /* % means moduo */

if that would not be enough, increase divider (2) and add more insert statements. The idea behind multiple inserts is to use less log space and reuse log space.

Or, if possible for You, use SSIS and have one source component with your select query and one destination component (don't forget to set batch size).

Answer 2

As @usr pointed out this is just too much data. Should have done the math up front. A square of a big number is a really big number. This is more for Azure Table Storage. This appears to work. Gave the check Filip.

DECLARE @sid int

DECLARE sID_cursor CURSOR FOR 
SELECT top 10 sID
FROM docSVsys
ORDER BY sID

OPEN sID_cursor

FETCH NEXT FROM sID_cursor
INTO @sID

WHILE @@FETCH_STATUS = 0
BEGIN
    PRINT @sID

    insert into sIDcrossMatch
    select @sid, docSVsys.sID, Abs(Checksum(NewId())) % 100 As RandomInteger 
    from docSVsys 
    where docSVsys.sID <> @sid
    order by docSVsys.sID;
    checkpoint;

    FETCH NEXT FROM sID_cursor
    INTO @sID

END 
CLOSE sID_cursor;
DEALLOCATE sID_cursor;