Defer insert data into a database

Question

I need to insert some records into a database at a time when it normally it's not being used, what are some approaches I can take? I have considered using the SQL Server Agent to run some stored procedures, or I could schedule a program to do the insertions too, ideally if the process takes more than 1 hour It should be able to stop and continue where it left off next time. Any suggestions on how can I do this cleanly, the technology to be used is .Net Core with Dapper, and the DB Server is MSSQLS 2016.

Answer 1

I would strongly advise you to avoid using scheduling tasks for low useage periods as a solution to long running jobs.

The problem with the approach is that these periods are not guarenteed to exist. Either you simply run out of hours in the day to schedule the tasks or your useage patterns change shrinking or ellimintaing the low periods.

Instead, implement a throttle so that the task can run at any time, but the speed of the task can be reduced so that it doesnt hog all the resources.

With a bit of clever programming you can monitor the speed of your database/api responses and dynamically throttle the application back as they slow down.