Inserting data each day into SQL Server from csv

Question

I need to insert large ammount of data each day into my SQL Server database. The data is inserted from the file, where rows represent data from this year, and some of these rows may be changed, while new rows are added to the file each day, so I need to check if some row is changed in the file and to update database and always to insert new rows.

So, what approach do you recommend (clear database/bulk insert, read line-by-line and insert in C#, ssis, etc.) ?

Answer 1

Based on your comments, I would just drop the table and reload the CSV every night using SSIS, then create fresh indexes as part of the nightly job.
If each CSV contains all of the relevant information, then this is the simplest way to go. No reason to fool around with update/merge logic that I can see.

Plus, given your aversion to SSIS, a straight table load with index creation should be very easy to implement in a C# script.

SSIS Route:

First, build the first load as demonstrated here.

Next, right click on the table is SSMS and generate the create script for that table.

Then, create an Execute SQL task in SSIS that runs before the load task. That SQL task will run the following 2 pieces of code drop table <your table name>, followed by the create table script you copied earlier.

Finally (and optionally), create an Execute SQL task that runs after the data load task that will create any needed indexes. Since I know nothing about your data, I'd recommend a nonclustered index that includes all of the columns that you use for parameters in your report, such as CREATE NONCLUSTERED INDEX IX_SalesPerson_SalesQuota_SalesYTD ON Sales.SalesPerson (SalesQuota, SalesYTD);

While not a perfectly tuned solution, it should suffice for what you are trying to do and be easy to maintain.

I may be able to add screenshots later.

Answer 2

There are different approach depends on how data is being used by application for the moment I would suggest following steps: Create schedule a job to upload data into Temp Table(s)

1. Bulk Insert into Temp Table
2. Insert New rows form Temp Table to Main Table
3. Update existing rows from Temp to Main Table

or You can delete existing rows by comparing Temp Table to Main Table and Then Build Insert every thing from Temp Table to Main Table.

Answer 3

Sounds like you need an "upsert". SQL server supports the MERGE statement that achieves this. You could feed your csv data into a temp table & then MERGE it into your destination table w/that syntax. Probably SSIS would let you set that up in a neat job.