Firebird on C#: FbCommand.ExecuteReader() performance

Question

I'm using C# (sharpdevelop) to access a firebird 2.5 server. This works really well. However, I have run into a performance problem. I have two tables, Members and Transactions which store who bought what when. To see who has not been active in a while I run the following compound query:

SELECT * FROM members M 
WHERE exists (select 1 from transactions WHERE taDateTime >= '30.06.2013' and Memberid = M.ID)

So, the Members-table has a primary index on field ID and the table transactions has indeces on both taDateTime and MemberID.

myConnection = new FbConnection(myConnectionString);
Connection.Open();

// property of a helper class, uses myConnection
// Then when I press the Search button:
// create the sql command, because the cut-off date is passed as a dbParam
IDbCommand aCmd = new FbCommand(sql, Connection);
IDataReader aReader = aCmd.ExecuteReader();

I now have a few 10'000 entries in members and several 100'000 entries in transactions. If I run this query in flamerobin, it takes about 2.6 sec to run. That's not thrilling, but sufficient. But If I run this query in C#, I need 3 minutes just to return from the call FbCommand.ExecuteReader(sql). I've not yet even looped over the result set, this is purely the call to ExecuteReader(). What does that do which takes so long?

I've tried several other things, sich as limiting the number of results by adding a ROWS or FIRST clause, both with no effect. So it's not the number of results which matters. I've tried to split the call into 2 queries: First find the DISTINCT(Transaction.MemberID) and then find the Members which fit, but there are too many IDs to sensibly pass into the second query. I've even programmed a loop which first finds the MemberIDs and then runs a "SELECT * FROM Members where ID = xx" for every result and that was faster than the single statement. So what is it which makes the call to ExecuteReader delay so long?

Please help, I'm really at a loss here.

Kind regards, Hannes

Answer 1

I should have clarified this by commenting instead of posting an answer, however, as a newbie I am not allowed to ask for more information (i.e. commenting)!!!

Quoting your line "To see who has not been active in a while I run the following compound query:", it appears to me that the correct query in this scenario should have been...

SELECT * FROM members M 
where M.ID not in (select distinct memberid from transactions WHERE taDateTime >= '30.06.2013')

The query you have written is going to give you people who are active. Also, it loads the db server with a processing of a few 10,000 * 100,000 records.

If you are indeed looking for people who are NOT active after a certain date, the query I gave you should give you at least 100 times faster results then previous one. I am curious to know the outcome, please let me know in any case. Thanks.

Answer 2

I agree with @Ravi that your query itself has some edges that might need some smoothing. I am not good enough in SQL to really anticipate what happens under the hood, but just from my gut feeling I'd say your statement looks "suspicious" to me...

But that still would not answer your question alone, otherwise the execution time in Flamerobin would be just as bad as when executed from your code. For that reason I expect something in you code to be the culprit. It would be great if if you could update your post with some more code.

Things to consider:

• Is the database connection already open? Opening a connection is quite costly performance wise.
• Do you have Connection Pooling activated? Re-using connections improves performance significantly.
• Are there other resource intensive tasks/ threads or running simultaneously? Or are there other read/ write accesses to the database that run at the same time when you are executing that statement? Possibly several accesses are waiting for each other to finish.
• Did you bind the query results to your UI? There are hundreds of potential performance bottle necks here.
• Also, if you have profiler at hand, have a look at the execution plan. Does the statement use the indices? Or does it perform a full table scan (or whatever strange things can happen)?
• Do you Prepare() your command before execcution? If you are firing it several times against the database this will also give you a performance gain.

And do not forget, due to the initializations and allocations that .NET code needs when it is executed by the JIT compiler, your statement might run faster when it is run a second time. Try this and run your query a second time (or multiple times) right after the first query. Does it make difference?

Answer 3

Thanks for all the input, guys! Googling some keywords in your suggestions made me analyse my queries in much more detail and thus find the error: My prog "secretly" added an "ORDER BY Member.Name" clause at the end of the query, which is the culprit. I must have been looking into my eyelids to miss that :) Anyway, I found one incredibly useful article for understanding and tuning queries in firebird and can only recommend it to anyone who's interested: http://conferences.embarcadero.com/article/32256. Kind regards, Hannes