Best way of constructing dynamic sql queries in C#/.NET3.5?

Question

A project I'm working on at the moment involves refactoring a C# Com Object which serves as a database access layer to some Sql 2005 databases.

The author of the existent code has built all the sql queries manually using a string and many if-statements to construct the fairly complex sql statement (~10 joins, >10 sub selects, ~15-25 where conditions and GroupBy's). The base table is always the same one, but the structure of joins, conditions and groupings depend on a set of parameters that are passed into my class/method.

Constructing the sql query like this does work but it obviously isn't a very elegant solution (and rather hard to read/understand and maintain as well)... I could just write a simple "querybuilder" myself but I am pretty sure that I am not the first one with this kind of problem, hence my questions:

• How do you construct your database queries?
• Does C# offer an easy way to dynamically build queries?

Answer 1

I used C# and Linq to do something similar to get log entries filtered on user input (see Conditional Linq Queries):

IQueryable<Log> matches = m_Locator.Logs;

// Users filter
if (usersFilter)
    matches = matches.Where(l => l.UserName == comboBoxUsers.Text);

 // Severity filter
 if (severityFilter)
     matches = matches.Where(l => l.Severity == comboBoxSeverity.Text);

 Logs = (from log in matches
         orderby log.EventTime descending
         select log).ToList();

Edit: The query isn't performed until .ToList() in the last statement.

Answer 2

Unless executiontime is really important, I would consider refactoring the business logic that (so often) tends to find its way down to the datalayer and into gazillion-long stored procs. In terms of maintainabillity, editabillity and appendabillity I always try to (as the C# programmer I am) lift code up to the businesslayer.

Trying to sort out someone elses 8000 line SQL Script is not my favorite task.

:)

//W

Answer 3

LINQ is the way to go.

Answer 4

This is the way I'd do it:

public IQueryable<ClientEntity> GetClients(Expression<Func<ClientModel, bool>> criteria)
    {
        return (
            from model in Context.Client.AsExpandable()
            where criteria.Invoke(model)
            select new Ibfx.AppServer.Imsdb.Entities.Client.ClientEntity()
            {
                Id = model.Id,
                ClientNumber = model.ClientNumber,
                NameFirst = model.NameFirst,
                //more propertie here

            }
        );
    }

The Expression parameter you pass in will be the dynamic query you'll build with the different WHERE clauses, JOINS, etc. This Expression will get Invoked at run time and give you what you need.

Here's a sample of how to call it:

public IQueryable<ClientEntity> GetClientsWithWebAccountId(int webAccountId)
    {
        var criteria = PredicateBuilder.True<ClientModel>();
        criteria = criteria.And(c => c.ClientWebAccount.WebAccountId.Equals(webAccountId));
        return GetClients(criteria);
    }

Answer 5

Its worth considering if you can implement as a parameterised strored procedure and optimise it in the database rather than dynamically generating the SQL via LINQ or an ORM at runtime. Often this will perform better. I know its a bit old fashioned but sometimes its the most effective approach.

Answer 6

I understand the potential of Linq but I have yet to see anyone try and do a Linq query of the complexity that Ben is suggesting

the fairly complex sql statement (~10 joins, >10 sub selects, ~15-25 where conditions and GroupBy's)

Does anyone have examples of large Linq queries, and any commentary on their manageability?

Answer 7

Linq to SQL together with System.Linq.Dynamic brings some nice possibilities.

I have posted a couple of sample code snippets here: http://blog.huagati.com/res/index.php/2008/06/23/application-architecture-part-2-data-access-layer-dynamic-linq

...and here: http://episteme.arstechnica.com/eve/forums/a/tpc/f/6330927813/m/717004553931?r=777003863931#777003863931

Answer 8

I'm coming at this late and have no chance for an upvote, but there's a great solution that I haven't seen considered: A combination of procedure/function with linq-to-object. Or to-xml or to-datatable I suppose.

I've been this in this exact situation, with a massive dynamically built query that was kindof an impressive achievement, but the complexity of which made for an upkeep nightmare. I had so many green comments to help the poor sap who had to come along later and understand it. I was in classic asp so I had few alternatives.

What I have done since is a combination of function/procedure and linq. Often the total complexity is less than the complexity of trying to do it one place. Pass some of the your criteria to the UDF, which becomes much more manageable. This gives you a manageable and understandable result-set. Apply your remaining distinctions using linq.

You can use the advantages of both:

• Reduce the total records as much as possible on the server; get as many crazy joins taken care of on the server. Databases are good at this stuff.
• Linq (to object etc.) isn't as powerful but is great at expressing complex criteria; so use it for various possible distinctions that add complexity to the code but that the db wouldn't be much better at handling. Operating on a reduced, normalized result set, linq can express complixity without much performance penalty.

How to decide which criteria to handle in the db and which with linq? Use your judgement. If you can efficiently handle complex db queries, you can handle this. Part art, part science.

Answer 9

You may want to consider LINQ or an O/R Mapper like this one: http://www.llblgen.com/

Answer 10

If using C# and .NET 3.5, with the addition of MS SQL Server then LINQ to SQL is definitely the way to go. If you are using anything other than that combination, I'd recommend an ORM route, such as nHibernate or Subsonic.

Answer 11

There a kind of experimental try at a QueryBuilder class at http://www.blackbeltcoder.com/Articles/strings/a-sql-querybuilder-class. Might be worth a look.

Answer 12

Check out http://sqlom.sourceforge.net. I think it does exactly what you are looking for.