How to work out optimum size for SaveChanges in EF4?

Question

I am reading a text file into a database through EF4. This file has over 600,000 rows in it and therefore speed is important.

If I call SaveChanges after creating each new entity object, then this process takes about 15 mins. If I call SaveChanges after creating 1024 objects, then it is down to 4 mins.

1024 was an arbitrary number I picked, it has no reference point.

However, I wondered if there WAS an optimum number of objects to load into my Entity Set before calling SaveChanges? And if so...how do you work it out (other than trial and error) ?

Answer 1

This is actually a really interesting issue, EF becomes much slower as the context gets very large. You can actually combat this and make drastic performance improvements by disabling AutoDetectChanges for the duration of your batch insert. In general however the more items you can include in a transaction in SQL the better.

Take a look at my post on EF performance here http://blog.staticvoid.co.nz/2012/03/entity-framework-comparative.html, and my post on how disabling AutoDetectChanges improves this here http://blog.staticvoid.co.nz/2012/05/entityframework-performance-and.html, these will also give you a good idea of how batch size affects performance.

Answer 2

Profile the application and see what's taking the time on the two you'ce chosen. That should geve you some good numbers to extrapolate from.

Personally I have to query why you are using EF for loading a text file - it's seems like gross overkill for something that should be easy to fire into a DB using BCP, or straight SQLCommands.