How often to call DataContext.SubmitChanges() for a large number of inserts?

Question

How many InsertOnSubmit should I call before calling SubmitChanges? I'm adding data from a web service that can return tens of thousands of records one record at a time. The wrapper class around the web service exposes the records a an IEnumberable collection to hide an elaborate chunking mechanism.

Are there guidelines on how many inserts I should accumulate before submitting them?

Answer 1

It also depends on the kind of data you need to insert. Sometimes I need to insert a lot of records, where I also need the ID of, to insert more records in another table.

Because the ID gets assigned when you submit changes to the database, I need to call SubmitChanges on specific times.

When this is not needed, I just submit them with a 1000 at once or so (depending on the total number of records I need to insert).

Maybe you could do some speed tests, which are best for you. Depending on the hardware, expectations of amount of records, etc.

Answer 2

Well i've done it with hundreds of thousands of records across multiple tables in the one go without a problem. In fact, whereas calling SubmitChanges() for every InsertOnSubmit() in a case like that would hours, just calling SubmitChanges() right at the end would reduce the time takes to insert that many records to a couple of minutes.

In the case i described above, the arrangement i had was for reporting tables with a header table, details table (which linked to header) and an atom table (which linked to details). For every header record, i had multiple detail tables which would then again be linked by multiple atom records. In some cases, i'd end up inserting gazillions of records and they'd all go in no problems with the single SubmitChanges() call at the end and it all performed very well.

Answer 3

For making changes to one field for 1000 records in batch vs 1 at a time: 1 record at a time takes 0.1989 sec/record vs batch which takes 0.0711 sec/record. So Batch is about 3x as fast in my tests. Note that the batch sizes vary in efficiency.

submitchanges() 1 record at a time
records sec sec/record
1000 198.95 0.1989

Batch submitchanges()
records sec sec/record % change
100 12.20 0.1220 133%
200 18.33 0.0916 122%
500 37.59 0.0752 106%
1,000 71.07 0.0711 103%
2,000 138.64 0.0693 102%
10,000 680.36 0.0680

Answer 4

There aren't really any "Guidelines" per-say. Id would say for efficiency you'd want to collect a bunch, maybe not 10k, but say 100? That would drastically reduce your DB Queries and shouldn't eat up too much ram caching them locally while you build up a transaction.

You should probably test with a couple different values and profile the performance (memory & speed) to find the optimal solution for your hardware though.