Moving data from staging table to production table uses up to 200GB of Log space

Question

I have multiple staging tables containing data that needs to be moved to production tables. The process of moving this data is done within a transaction to make it possible to rollback everything in case the process fails. The most log consuming operations are:

• Deleting old data from production tables
• Inserting new data into production tables

The query used to delete the old data is:

DELETE FROM PROD_TABLE WHERE CONDITION

A simple truncate is not possible since not all the data can be deleted.

The data to be deleted and the data to be moved is about 200 Million records.

Is there anyway to improve the process to reduce the log consomption.

Answer 1

You might like to look at partition switching - the staging table could be switched in to the production table, and the production table switched out to a history table. https://docs.microsoft.com/en-us/sql/t-sql/statements/alter-table-transact-sql?view=sql-server-ver15#c-switching-partitions-between-tables gives an example of a switch out. There are quite a few blogs about the technique, look up a few and see if they match what you are trying to do.

A system I used to work on had a monthly data load. The data was staged to a table, which was then switched in to the live table. The live table had a partition per month. It was very quick as I think it's a metadata operation. HTH