Why does the default GetById() for a DAL2 repository in DNN execute slowly?

Question

I am having a problem where the default Get() function for a DAL2 repository is taking several seconds to return results. When I query the database directly for the record with SQL Server Management Studio the query returns quickly. The function is a basic function and looks like this:

public MyThing Get(string primaryKey)
{
    MyThing myThing;
    using (IDataContext ctx = DataContext.Instance(“DATABASECONN”))
    {
        var rep = ctx.GetRepository<MyThing>();
        myThing = rep.GetById(primaryKey);
    }
    return myThing;
}

The table has 800K+ entries.

The function has the same execution time in subsequent executions even though the cache setting is enabled.

Why is this functioning slowly and how can I speed it up?

Answer 1

The Investigation

I ran SQL Profiler on the database while debugging the code. It showed that the first time the GetById() function is called, a query to select all records from the database table is executed. The primary key is not passed to the database. The next time the GetById() function is executed, the database is not queried. The results are returned from the cached value.

The single record returned from the GetById() function must therefore be the result of some search through the result set of all records in the table. The details of how this function operates and retrieves the record are in compiled code (the source for this compiled code is likely available but I have not yet investigated it).

I looked at the memory usage during the execution of this code to make sure IIS had an adequate amount. The settings in IIS were set to allow the application pool to use the available memory. While debugging the memory usage for the w3wp.exe process fluctuated between .75 and 1.3 GB. There was plenty of extra RAM available so the site could use more if needed.

The highlights of these experiments are:

• The first time a GetById() function hits a database table, it will select all records from the table.
• Cache is functioning as expected so repeated queries are not hitting the database.
• Retrieving results from Cache is taking several seconds no matter how much memory is available to IIS.

The Solution

To solve this problem I altered my approach. When querying the database directly using the primary key during the SELECT, the query returned quickly. This lead me to use the Find() function of the DAL2 repository so I could specify the conditions of the SELECT myself.
The function I created looks like this:

public MyThing Get(string primaryKey)
{
    MyThing myThing = (MyThing)DataCache.GetCache(primaryKey);
    if (myThing != null)   
        return myThing;

    IList<MyThing> myThings;
    using (IDataContext ctx = DataContext.Instance(“DATABASECONN”))
    {
        var rep = ctx.GetRepository<MyThing>();
        myThings = (IList<MyThing>)rep.Find("WHERE myPrimaryKeyFieldName = @0", primaryKey);
        if (myThings.Count > 0)
            DataCache.SetCache(primaryKey, myThings[0]);
    }
    return myThings.Count > 0 ? myThings[0] : null;
}

The function above uses the Find() function and as a result caching is not implemented by DAL2. To help out with this I have included a custom caching technique. Using SQL Profiler to investigate this function showed that repeated calls to the function using the same primaryKey value did not hit the database, so cache was behaving as expected.

Cache used in this manner does not slow the execution time like it does when using GetById(). DateTimes placed within the function, used as timers, show the following execution times. When the database was hit, 00:00:00.1 or less. When the cache was hit the execution time was 0. This was not tested on a large scale but execution times were consistent in the tests I ran.

Conclusion

The tests above and the solution show that the GetById() in DAL2 should be used with care. It is not necessarily the best solution to retrieve a record depending on the size of the database. If data retrievals are executing slowly, explore alternative methods such as the method suggested in the solution.

It would be helpful to figure out the exact number of records in the table where GetById() is no longer the best solution. This number likely varies depending on the the size of the data in the object. It is likely computable but there is currently no equation so it will need to be figured out on a case by case basis.

If anyone else has extra information on this topic, please add it to the discussion.