Update a Db from a text file

Question

We have two tables in our database that need to be updated using an extraction of another source. This extraction provides us a text-file.

The first insert was easy thanks to Talend but now we have another problem.

We don't want to do a simple insert or update but we want to compare both sources and the give the choice to the user to select which values are good for each entry.

For the moment the only solution found is to insert the new content in another table and the to compare each entry thanks to the program, then we display the entries that have had modifications.

The problem is that we have 1500+ entries, and this solution has very low performances, so we're looking for a new one.

Would there be a solution like extrading from our db into a text file, then compare both files automatically and gerenate a third file including only the modifications. This third one would be easier to load in the program and to be analyzed.

Thanks for your help

Answer 1

Since there are only a few records (1500+) in the database:

• load all records from the database into memory,
• start reading every line in the updated input file
• see if there is an exact match with the records in memory
• if there is not, insert the record into the database with a link to the original db record

For your selects:

• If the 'link' column is empty it is an 'actual record'
• If there is a value in the link column, you can use that to select the records that need to be compared by the user

When your database becomes too large to load everything into memory, you'll have to switch to batch processing (using PL/SQL or T-SQL or...) or change the process somewhat.