Converting Linq2SQL simple queries to CompiledQueries to increase app performance

Question

I am writing a Silverlight for Windows Phone (SDK 7.1) app and I am displaying data from a CompactSQL DB in a LongListSelectorcontrol from the Silverlight Toolkit for Windows Phone.

Once the list becomes about 150 items long, The app really slows down loading data, navigating to and from pages and animations fail to display (I know using a background thread would help with freeing up the UI thread for animations).

I currently have three queries that I use constantly - everytime the data from LongListSelector is updated or the page is NavigatedTo. I have converted MoviesByTitle into a CompiledQuery and that has helped quite a lot, so I was looking to do the same for my other two queries(groupedMovies and LongListSelector.ItemSource of type List<Group<Movie>>), however I cannot seem to figure out the correct syntax.

Any suggestions on how I might make these queries more efficient - Through the use of CompiledQuery or otherwise?

MoviesByTitle is in another Class called Queries

public static Func<MovieDBContext, IOrderedQueryable<Movies>> MoviesByTitle = CompiledQuery.Compile((MovieDBContext db) => from m in db.Movies orderby m.Title,m.Year select m);

Fields in MainPage

private static List<String> characters = new List<String> { "#", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z" };
public static List<Group<Movies>> emptyGroups = new List<Group<Movies>>();

inside a LoadDB() method in MainPage - this method is called in OnNavigatedTo and in several other places when the DB is updated.

//Populates the 'empty' Groups of Movies objects only once.
if (emptyGroups.Count < 1)
{
    characters.ForEach(x => emptyGroups.Add(new Group<Movies>(x, new List<Movies>())));
}

IEnumerable<Movies> query = Queries.MoviesByTitle(App.db);

//Groups the objects 
IEnumerable<Group<Movies>> groupedMovies = (from t in query
                                            group t by t.GroupHeader into grp
                                            orderby grp.Key
                                            select new Group<Movies>(grp.Key.ToString(), grp));

//Joins the 'empty' group and groupedMovies together for the LongListSelector control
moviesLongList.ItemsSource = (from t in groupedMovies.AsEnumerable().Union(emptyGroups)
                              orderby t.Title
                              select t).ToList();

GroupHeader is a property of Movies and an entity in the DB

[Column(CanBeNull=true, UpdateCheck = UpdateCheck.Never)]
public char GroupHeader
    {
        get
        {
            char l;
            //For Alphabetized Grouping, titles do NOT start with "The ..."
            if (this.Title.ToLowerInvariant().StartsWith("the "))
            {
                l = this.Title.ToLowerInvariant()[4];
            }
            else
            {
                l = this.Title.ToLower()[0];
            }
            if (l >= 'a' && l <= 'z')
            {
                return l;
            }
            return '#';
        }
        set { }
    }

The Group class is as follows

public class Group<T> : IEnumerable<T>
{
    public Group(string name, IEnumerable<T> items)
    {
        this.Title = name;
        this.Items = new List<T>(items);
    }

    public string Title
    {
        get;
        set;
    }

    public IList<T> Items
    {
        get;
        set;
    }
    ...
}

Answer 1

I assume that GroupHeader is an entity stored in the DB with 1-n relationship to the Movie entity.

First of all, I don't see 3 DB queries here. A LINQ expression is not always a DB query (e.g. there's LINQ to Objects). Sometimes determining what really is going on is quite challenging. The best friend in such cases is a DB profiler tool or the IntelliTrace - they show what queries are being executed on the DB at run time.

As far as I understand the code, you actually have 1+N queries: the first is MoviesByTitle, and then you have N queries in the expression that gets the movies grouped by their headers. It's N instead of 1 because you cast query to IEnumerable<> which makes it no longer a query but a simple CLR object, which is simply iterated in a foreach-like loop sending queries to the DB each time it needs a GroupHeader entity (it's an entity, isn't it?).

Try to combine 2 queries into one. You might even not need to use CompiledQuery. Here's an approximate code:

// I left this without changes
if (emptyGroups.Count < 1)           
{           
    characters.ForEach(x => emptyGroups.Add(new Group<Movies>(x, new List<Movies>())));           
} 

// I put var here, but it actually is an IQueryable<Movie>
var query = from m in App.db.Movies orderby m.Title, m.Year select m;

// Here var is IQueryable<anonymous type>, I just can't use anything else but var here
var groupedMoviesQuery = from t in query
                    group t by t.GroupHeader into grp 
                    orderby grp.Key 
                    select new
                    {
                       Movies = grp,
                       Header = grp.Key
                    }; 

// I use AsEnumerable to mark that what goes after AsEnumerable is to be executed on the client
IEnumerable<Group<Movie>> groupedMovies = groupedMoviesQuery.AsEnumerable()
                                            .Select(x => new Group<Movie>(x.Header, x.Movies))
                                            .ToList();

//No changes here
moviesLongList.ItemsSource = (from t in groupedMovies.AsEnumerable().Union(emptyGroups)            
                              orderby t.Title            
                              select t).ToList(); 

This code should work way better. What I actually did is that I turned your query from IEnumerable which is an iterate-only CLR object to an IQueryable which can be further wrapped into a more complex query. Now there's only one query which gets all movies grouped by headers. It should be fast.

I would introduce even more improvements to the code to make it work even faster:

1. You use Union of entities read from the DB and of some default list. You order it afterwards. You can safely remove all other orderings from your Linq2Sql code ('query' and 'groupedMoviesQuery')
2. Grouping seems to be not the most efficient here as compared to a join. Why don't you just query GroupHeaders including their related Movies? That should produce a JOIN in the db query which should be more efficient than a GROUP BY.
3. If you still have issues with performance, you can turn the query into a compiled query.

I'll show an example of a compiled query for you original logic with optimization of the first item of the list above:

class Result
{
  public GroupHeader Header {get;set;}
  public IEnumerable<Movie> Movies {get;set;}
}

public static Func<MovieDBContext, IQueryable<Result>> GetGroupHeadersWithMovies =
  CompiledQuery.Compile((MovieDBContext x) => 
      from m in x.Movies
      group m by m.GroupHeader into grp 
      select new Result
      {
        Movies = grp,
        Header = grp.Key
      });