Socket Connection Data Insert

Question

I have been working through a high-performance application where I have identified a bottleneck. The bottleneck is actually when the application must insert messages from a socket, it will record them in a database. The trouble is the speed at which the messages come through are so fast that the application cannot catch up due to poor architectural planning. Clearly, insertion needs to be improved.

So here's the question: What is the most optimal format for performing inserts of individual records?

I have looked over at this link on high performance inserts revision 4 and found that doing something of this nature would require an extensive restructure in the application compared to its current attempt at concatenating a string together such as this:

INSERT INTO table (col1,col2,col3) VALUES (val1,val2,val3),(val4,val5,val6),(val7,val8,val9)

Does anybody have any have an insight over which is better, datatable insertion vs string concatenation?

Answer 1

Well, thanks for anybody who might have read this and considered what to do but I decided to try and restructure the application because I am running out of time.

Winner by a mile: DataTable insertion as stored procedure parameter

Concatenating strings only allowed me to create and send about 750 records at a time when it caused the application to slow down enough to get overwhelmed and throw an exception. Using a datatable insertion described in the link allowed me to safely go way above 3,000 records at a time. Knock on wood, my application has not crashed yet during testing. It seems like I can drink from a fire hose.

Just to put some stats around it, my table had 18 million records before I revised the application. In approximately 1 hour, I have inserted 1.3 million more records coming from the socket one at a time. It feels like the method could take a lot more than I am currently testing it with.

Answer 2

first disable all transaction isolation repeatable reads etc., maybe put this table into separate db and then make it single user, so no overhead for sql server, if not then make in your stored procedure datastream and use bcp mechanics (just google) and load data into table from this stream, it is crazy fast. BR Alex from Vilnius :)