Database Log file growing , after ruining Bulk inserts in C#

Question

I am running a C# application which bulk inserts about half a million records per table into a sql Database.

After processing a file , the log file grows to 4 gb! I have to back up every night as a result.

Is there away I can improve this? minimize growth of the log file ?

 using (SqlBulkCopy bulkCopy = new SqlBulkCopy(ConnectionString))
            {
                bulkCopy.BulkCopyTimeout = 700; // in seconds
                bulkCopy.DestinationTableName = TableName;
                bulkCopy.WriteToServer(DT);
            }

Answer 1

Assuming your database is configured using the "Full" or "Bulk Logged" recovery modes, space in the transaction log can only be reused once a transaction log backup has been performed. If the log hasn't been backed up and there's no free space to log changes, then the transaction log is either expanded automatically (if configured to do so), or the transaction fails. It's therefore not uncommon to back up the transaction log frequently (say, every 15 mins) to avoid excessive transaction log growth.

If you're using the "Full" recovery mode, you may be able to switch to using the "Bulk Logged" mode, which reduces the amount of space taken by bulk copy operations.

Alternatively, for non-production scenarios where you may be less concerned by recoverability between full backups, then you can switch the recovery mode to "Simple". This does not require any transaction log backups, and as a result, the transaction log should rarely grow.

More details about each recovery mode can be found here.

Answer 2

You are adding data in one transaction currently. If your database is in Simple mode the transaction log will grow one time and stay at that size as it reuses the log file. If you broke this down into a few transactions then you could have a slower insert but also a smaller transaction log when in simple mode.

If this is for a production database where the you care about backups you are able to run hourly or even possibly more regular transaction log backups which will enable a higher resolution for recovery of data.

To provide any accurate information about what to do need to know if this is a throwaway database or one where you keep the data long term and require it to be backed up.

Edit: Since you have this as a production database I would recommend doing transaction backups as regularly as you require. In the case of our business we do them every 5 minutes. Combining regular transaction backups with splitting your insert into multiple transactions will keep a small transaction log and allow granular restore points in the case of issue. http://technet.microsoft.com/en-us/library/ms191284(v=sql.105).aspx

Answer 3

This is just a guess:

Your database may be in full recovery mode which logs everything.

Try changing it to bulk-logged which reduces log space usage by using minimal logging for most bulk operations.

This is the only thing I can think of.