loading fact table using pentaho data integation

Question

I am using pentaho DI to insert data into fact table . Thing is the table from which I am populating my fact table contains 10000 reccords and changes frequently . Using database lookups and insert update I am able to load my fact table correctly once . But when new records are added to my souce table(say it becomes 15000) and I am again inserting records into fact table then these 15000 recods are again added to my fact table . What I want is to add new 5000 records that do not exist in fact table .Please suggest me on what transformations I need to perform to acheive this .

Answer 1

try doing an upsertion instead insertion (if the row exists then update , if not insert)

Answer 2

You can use some DB function.

In SQL Server 2008, there is a merge sql that solve this type problem.

It is a example as follows in SQL Server 2008:

MERGE Production.UnitMeasure AS target USING (SELECT @UnitMeasureCode, @Name) AS source (UnitMeasureCode, Name) ON (target.UnitMeasureCode = source.UnitMeasureCode) WHEN MATCHED THEN UPDATE SET Name = source.Name WHEN NOT MATCHED THEN
INSERT (UnitMeasureCode, Name) VALUES (source.UnitMeasureCode, source.Name) OUTPUT deleted., $action, inserted. INTO #MyTempTable;