Batching DB commands in Entity Framework 4.0

Question

My current project requires everyday synchronization with external system. The synchronization is based on complex import file structure which is parsed and processed with extensive business logic. Due to the business logic we decided to make this in .NET code and reuse existing BL components instead of writting the same logic in stored procedures or integration services.

The BL layer sits on top of EF 4.0 data access layer. Current implementation process the import batch, fills all changes into ObjectContext and executes SaveChanges in transaction. When I check SQL profiler I see that EF executes each entity change as single SQL command (with its own round trip to DB). Moreover it looks like these commands are executed fully sequentially. So I have up to 100.000 roundtrips to database for initial import and between 10.000 - 50.000 roundtrips to database for daily synchronization.

Is it possible to batch insert/update/delete commands somehow by EF itself or by some provider / extension?

Answer 1

No, it can't be done (yes - i weep also).

EF does not support batch operations, LINQ-SQL had (has) the same problem.

You've got a few options:

1. Stored Procedures
2. Classic ADO.NET or EntitySQL
3. Triggers

I've gone with option 1 and 3 in the past.

The problem with all three approaches is you lose the EF abstraction, the internal graph (optimistic concurrency), and your brought back to the world of native SQL.

Answer 2

Be aware of this project: magiq.codeplex.com

It brings batch operations to linq-to-sql and we are working in EntityFramework support by now.

Cheers

Answer 3

There are some workarounds in SQL Server:

• Batch Inserts are described in the Entity Framework Bulk Copy article

• Batch Updates (and Deletes) are described in the Multiple entity updates with Entity Framework – EF Fetch Updates article.

In case you are interested in Oracle, MySQL, POstgreSQL, or SQLite, you can use the latest Devart dotConnect providers. The BatchUpdates functionality is already integrated in the SaveChanges method in the latest versions of these providers.

Answer 4

Here's a way that allows you to use your code first POCOs and is fast. Sped up a bulk insert from > 1 hour to ~5 seconds.

SqlBulkCopy for Generic List (useful for Entity Framework & NHibernate).

Answer 5

Just sharing with you a GitHub project just for that, as of now, it supports Bulk insert/update/delete for Sql server transparently using SqlBulkCopy. https://github.com/MHanafy/EntityExtensions There're other goodies as well, and hopefully, It will be extended to do more down the track. Using it is as simple as

var insertsAndupdates = new List<object>();
var deletes = new List<object>();
context.BulkUpdate(insertsAndupdates, deletes);

Hope it helps!