SSIS: Only Update DB not entire list

Question

i have a really simple question what is one of the ways to only fill a DB with data that is not already in the Table ?

I am currently trying to implement a Project which gets data from a SharePoint List and stores it in a Table on my DataWarehouse, and i had to realise that if run more than once, it keeps filling the table with data it already has!

Answer 1

Between your source and your destination, you want to evaluate whether there is a match in the destination (and possibly determine whether there have been changes between the values).

The SSIS tool you're looking for is the Lookup Transformation. More generally speaking, you're performing an incremental load and Andy Leonard has a great series called the Stairway to Integration Services that covers how to handle this at a basic to complex level based on your needs.

Answer 2

In plain english, you want a query like this:

insert into mytable
(fields)
select values
where they are not already in the table

There are several ways to do this. Here are a couple

where (select count(*)
from mytable
where whatever) = 0

where not exists
(subquery goes here)

where somefield in
(select somefield from staging_table
except 
select somefield from mytable)