TClientDataSet and big insert

Question

In my application I use TADOQuery with select (MSSQL) and linked with it TClientDataSet. I have to insert about million records and ApplyUpdates.

So what I see in the SQL Server Profiler? I see that for each inserted row we have 3 queries: sp_prepare of insert script, sp_execute it with some values and sp_unprepare.

I want just to prepare sql once for all of the records before insert and unprepare it after. How can I do it?

Added after:

In the query I have a script for the stored procedure execution:

tmpQuery := DefineQuery(FConnection, [
  'exec up_getOperatorDataSet ',
  '  @tablename     = :tablename, ',
  '  @operator      = :operator, ',
  '  @forappend     = :forappend, ',
  '  @withlinksonly = :withlinksonly, ',
  '  @ids           = :ids '
], [
  Param(ftString, sTableName),
  Param(ftInteger, FOperatorId),
  Param(ftBoolean, opForAppendOnly in OpenParams),
  Param(ftBoolean, opOnlyWithModelLinks in OpenParams),
  Param(ftString, sIds)
], Result);

It selects all of the fields from table sTableName with some parameters.

Example of inserting from profiler:

step 1:

declare @p1 int
set @p1=486
exec sp_prepare @p1 output,N'@P1 int,@P2 int,@P3 datetime,@P4 int,@P5 int,@P6 int,@P7 int,@P8 int,@P9 varchar(128),@P10 bit,@P11 numeric(19,4),@P12 smallint,@P13 smallint,@P14 smallint,@P15 smallint',N'insert into parser_prices
  (operator_id, request_id, date, nights, model_hotel_id, model_meal_id, model_room_id, model_htplace_id, spo, hotelstop, price, frout_econom, frout_business, frback_econom, frback_business)
values
  (@P1, @P2, @P3, @P4, @P5, @P6, @P7, @P8, @P9, @P10, @P11, @P12, @P13, @P14, @P15)
',1
select @p1

step 2:

exec sp_execute 486,21,2000450,'2009-12-04 00:00:00',14,2118,22,-9555,18,'2009-10.MSK.Bali.13.10.09-27.03.10',0,15530.0000,3,3,3,3

step 3:

exec sp_unprepare 486

and it is for all of the new rows.

Answer 1

The answer was in provider used in TADOConnection. Switched from MSDASQL to SQLOLEDB and all is right now, without any additional queries.

Answer 2

Because you're calling a stored procedure, and not an inline query in your code, then SQL Server is treating each call to the stored proc as a separate call, and so is preparing it and unpreparing it every time. I'm not sure if there is a way around this.

If whatever is happening within the stored proc can be done from a query in your code, then you could use a structure like this that would only prepare the SQL statement the first time:

{Prepare the insert query}
ADOQuery1.SQL.Append('INSERT INTO Tablename');
ADOQuery1.SQL.Append('(StringField1, IntField2)');             {repeat as necessary}
ADOQuery1.SQL.Append('VALUES (:sFieldValue1, :sFieldValue2)'); {repeat as necessary}
ADOQuery1.SQL.Prepare;

{In a For, While, Repeat loop, use:}
ADOQuery1.ParamByName('sFieldValue1').AsString := 'Value for field 1';
ADOQuery1.ParamByName('sFieldValue2').AsInteger := 2;
ADOQuery1.ExecSQL;

Apologies if I've not quite got the property and method names right for the ADOQuery component, I'm not on my Delphi PC at the moment and I don't usually use the TADO components, but the concept still applies as this is a TDataSet concept.

Answer 3

Thoughts...

1. You don't need to prepare a stored procedure call. In effect, it's already prepared. You can swicth it off in most client implementations.

2. You might not be able to do it for a million rows in one go. You have a batch size limit (eg a single DB call) of 256 MB (assuming default 4k network packets).

3. In other client implementations you can set a "batch size" (different concept to point 2), say 10,000, so you'd only make 100 calls not one million.

Answer 4

I think that the other answers may be good at helping to tune the performance, although I suppose what method you use to access the TClientDataSet doesn't really matter, since either way, the actual database update is separate (and automatically generated).

If the update works as a separate prepare for each line, then it seems that was a poor design choice by Borland, as it's obvious that updating many rows will require the same query, only with different parameters each time.

On the other hand, TClientDataSet was meant for in-memory databases, which means relatively small databases. Using something like a million rows probably goes beyond the intended use case.

On the other hand, swapping out the ClientDataSet from your application at this point might be troublesome. I would say that for the performance sensitive part of your application, I would track the modified rows myself and write a manual update routine using the routines mentioned above. Besides that, you could try modifying the TClientDataSet's source code to make it more efficient, or sub-classing it and overriding the method which applies the changes.

(Personally, I use SQLite3 for storage in my programs, so ClientDataSet is of little use and I haven't played with it much).