Insert List<> and check for duplicates

Question

im trying to build an application that reads news RSS feeds, and then puts the elements in a database. From which i later on can add statistics to each of the topics "titles".

I figured the best way to go at this is by make a script that parse the xml-data into a List, and then insert this list into database, and which it at the same time checks for duplicates?

Or do you have any smarter way?

Main question, How do I insert a List to database and at the same time prevent duplications? Thank you

Answer 1

If this is how you want to do it, instead of a List, look into using a Hashset. If there are a lot of elements in your news feed, a hashset can tell you quickly if it exists in the collection already or not. In pseudo code it would basically be:

HashSet.Add(Element);

Duplicate records are automatically ignored (Add will return true if the item is added, and false if it already existed).