Slow VBA-SQL (250 Sec for 620 Inserts), Network or Code?

Question

So I have written an application in VBA(Excel) to interface with a Firebird Datbase. The database used to run locally but have moved it onto a hosted solution which I access over VPN. (I have a 25/4 Mb internet connection at home)

Now when it ran locally my code was sloppy and I used generated SQL statements where I would concatenate the variables into an sqlstring and then execute. I.e: "INSERT INTO BLA..& VALUE1 & VALUE2" etc.

This worked fine the, but when moved to the hosted solution, was painfully slow, and for 623 inserts took 450 seconds.

Ok so I realized I didn't do it right the first time, so I rewrote the code as a paramterized sql query, the code is below.

Set adoComm = New ADODB.Command
Set adoConn = New ADODB.Connection
Call clientcleancall
mytimer = Timer
Dim sconnect As String

sconnect = "Provider=MSDASQL.1;DSN=SOSARemoteTesting;auto_commit=true"

Dim Conn As New ADODB.Connection

adoConn.Open sconnect
adoComm.ActiveConnection = adoConn
Application.ScreenUpdating = False


adoComm.Parameters.Append adoComm.CreateParameter("p1", adChar, adParamInput, 40)
adoComm.Parameters.Append adoComm.CreateParameter("p2", adChar, adParamInput, 40)
adoComm.Parameters.Append adoComm.CreateParameter("p3", adChar, adParamInput, 35)
adoComm.Parameters.Append adoComm.CreateParameter("p4", adChar, adParamInput, 20)
adoComm.Parameters.Append adoComm.CreateParameter("p5", adChar, adParamInput, 20)
adoComm.Parameters.Append adoComm.CreateParameter("p6", adInteger, adParamInput, 50)
adoComm.Parameters.Append adoComm.CreateParameter("p7", adChar, adParamInput, 60)
adoComm.Parameters.Append adoComm.CreateParameter("p8", adChar, adParamInput, 60)
adoComm.Parameters.Append adoComm.CreateParameter("p9", adChar, adParamInput, 60)
adoComm.Parameters.Append adoComm.CreateParameter("p10", adChar, adParamInput, 60)
adoComm.Parameters.Append adoComm.CreateParameter("p11", adChar, adParamInput, 10)
adoComm.Parameters.Append adoComm.CreateParameter("p12", adChar, adParamInput, 100)

Dim s As String

s = "INSERT INTO HARDWARE_ASSETS (HW_SERIAL, HW_NAME, HW_MACH_ID, HW_CUST, HW_INSTALL," & _
"HW_INSTALL_DATE, HW_LOCATION, HW_STATUS, HW_BILLABLE, HW_OWNER, HW_VIRTUAL, HW_SOURCE) " & _
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

adoComm.CommandText = s

For lrow = 3 To 624
      'flex1.Row = lRow
      'flex1.Col = 0
        Cells(lrow, 1).Select
        adoComm.Parameters("P1").value = Selection
        adoComm.Parameters("P2").value = Selection.Offset(0, 1).value
        adoComm.Parameters("P3").value = Selection.Offset(0, 2).value
        adoComm.Parameters("P4").value = Selection.Offset(0, 3).value
        'adoComm.Parameters("P5").AppendChunk (Mid(GetString(flex1.Text), 1, 20000))
        adoComm.Parameters("P5").value = Selection.Offset(0, 4).value
        adoComm.Parameters("P6").value = Selection.Offset(0, 5).value
        adoComm.Parameters("P7").value = Selection.Offset(0, 6).value
        adoComm.Parameters("P8").value = Selection.Offset(0, 7).value
        adoComm.Parameters("P9").value = Selection.Offset(0, 8).value
        adoComm.Parameters("P10").value = Selection.Offset(0, 9).value
        adoComm.Parameters("P11").value = Selection.Offset(0, 10).value
        adoComm.Parameters("P12").value = Selection.Offset(0, 11).value
        adoComm.Execute s, , adExecuteNoRecords

Next lrow

    bcommited = True

adoConn.Close
Application.ScreenUpdating = True
MsgBox (Timer - mytimer)

Now a BIG improvement in my execution time, but still 621 Inserts requires 250 seconds.

To access the server I must authenticate through a firewall, on top of the VPN. Some trouble shooting I have done is pinging the server, and the first packet always takes ~200ms, then the rest take about 100. Average is usually around 150ms. The firebird database is used by large organizations, with some as big a Terabyte, used in production, so I don't think the file-based nature of the database is the culprit.

My concrete questions: 1) How long should my code take to execute, and is the current 250 too long? 2) Is there anything else I can do to streamline my code? 3) If 1 & 2 are not the culprit I'd like to discuss things with my network admin as think the high ping times may be compounding the problem?

Thanks for your help .

Answer 1

COM ADO can be extremely fast. You're doing exactly what you should to get the most performance: reusing the same Command object and supplying the values with parameters. 621 inserts on an enterprise database, even with 12 parameters, should take on the order of a second, or several. So the low performance is almost certainly not due to your code.

Try three things to track down the bottle neck:

1. Comment out only the line:

   adoComm.Execute s, , adExecuteNoRecords
   

   Thus you will just create the command and its parameters and then zip through your spreadsheet assigning the values to the parameters. This will tell you how long it takes to read and load your local values. Excel can sometimes be ridiculously slow at enumerating its own native Range objects, but I don't know if that's an issue here untill you test it.

2. In a separate test sub, open a connection to the FireBird database, create a simple command like SELECT 1 FROM rdb$database¹ and execute the command 621 times. This will tell you how long it takes just to contact the server and wait for a response.

3. Repeat #2, but add 12 parameters to the query. Just set them once at the beginning. This will tell you how much additional time it takes to transfer the parameter values accross the wire. The SQL would be something like¹:

   SELECT ? AS P1, ? AS P3, ? AS P3, ? AS P4 .. FROM rdb$database
   

¹ I'm not familiar with FireBird SQL, but this forum entry and this official FAQ suggested that FireBird requires the 'magic' table name rdb$database for singleton selects...maybe you (the OP) already knew that

Answer 2

After working through this, the network was a contributing factor. With ping times as high as 200 ms, even the parameterized queries took long.

The solution for me was to use a Block insert.

Execute BLOCK as begin
Full insert;
Full insert;
Full insert;
end

Runtime for 621 inserts is now 5 seconds. Much more managable and realistic. The only limitiation is there is a character limit of ~58k. Which means you have to modify your loop to do only 100 or so inserts per block.