Updating a SQL database table with a webservice

Question

I have an issue and I have looked long and hard over the Internet for an answer but cant find anything.

I have a little app that sucks in a web service. It then passes this web services results another applications via its own web service and stores the request in a table.

What I am trying to do is quickly import the results to a table from the data set.

Option 1 is that I loop through all the rows in the data set and use an insert on each. The issue with this is it is slow and will slow down the response of the little apps web service.

Option 2 is to bulk upload the data set into sql. The bad news for me is I have no idea of how to do this!! Can any one help?

Answer 1

You can use SqlBulkCopy. A simple SqlBulkCopy might look like:

DataTable dtMyData = ... retrieve records from WebService
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connectionString)) {
    bulkCopy.BulkCopyTimeout = 120; // timeout in seconds, default is 30
    bulkCopy.DestinationTableName = "MyTable"; 
    bulkCopy.WriteToServer(dtMyData);
}

If you are processing a great deal of data you may also want to set the BatchSize property.

Answer 2

It sounds like a mighty busy little web service.

You might instead (or even in addition) consider making it asynchronous or multi-threaded. Otherwise you've just built in the coupling between the two web apps that you may have split them to avoid in the first place.

Answer 3

Using a bulk load can be a pain for various reasons, starting with that how you do it is specific to a particular DBMS. (The next thing that usually gets you is where the file with the data needs to be located.)

However, you can often improve performance dramatically in a portable way merely by doing all of your INSERTs in a single transaction:

BEGIN TRANSACTION
INSERT ...
INSERT ...
...
COMMIT TRANSACTION

Answer 4

If time is on your side, you could look into using SQL Server Service Broker technology however there is initially quite a steep learning curve.

http://msdn.microsoft.com/en-us/library/ms166043(SQL.90).aspx

You can create a web service that fires say a stored procedure to insert the results of your process.