Creating complex XPQuery - LINQ to SQL with nested lists

Question

any hint on what's wrong with the below query?

return new ItemPricesViewModel()
            {
                Source = (from o in XpoSession.Query<PRICE>()
                          select new ItemPriceViewModel()
                          {
                              ID = o.ITEM_ID.ITEM_ID,
                              ItemCod = o.ITEM_ID.ITEM_COD,
                              ItemModifier = o.ITEM_MODIFIER_ID.ITEM_MODIFIER_COD,
                              ItemName = o.ITEM_ID.ITEM_COD,
                              ItemID = o.ITEM_ID.ITEM_ID,
                              ItemModifierID = o.ITEM_MODIFIER_ID.ITEM_MODIFIER_ID,
                              ItemPrices = (from d in o
                                                where d.ITEM_ID.ITEM_ID == o.ITEM_ID.ITEM_ID && d.ITEM_MODIFIER_ID.ITEM_MODIFIER_ID == o.ITEM_MODIFIER_ID.ITEM_MODIFIER_ID
                                                select new Price()
                                                {
                                                    ID = o.PRICE_ID,
                                                    PriceList = o.PRICELIST_ID.PRICELIST_,
                                                    Price = o.PRICE_

                                                }).ToList()
                          }).ToList()
            };

1. o in subquery is in read and I got the message "Could not find an implementation of the query pattern for source type . 'Where' not found."
2. I would like to have distinct ItemID, ItemModifier: should I create a custom IEqualityComparer to do it?

Thank you!

Answer 1

Probable cause

In general, XPO does not support "free joins" in most of the cases. It was explicitelly written somewhere in their knowledgebase or Q/A site. If I hit that article again, I'll include a link to it.

In your original code example, you were trying to perform a "free join" in the INNER query. The 'WHERE' clause was doing a join-by-key, probably navigational, but also it contained an extra filter by "modifier" which probably is not a part of the definition of the relation.

Moreover, the query tried to reuse the IQueryable<PRICE> o in the inner query - what actually seems somewhat supported by XPO - but if you ever add any prefiltering ('where') to the toplevel 'o', it would have high odds of breaking again.

The docs state that XPO supports only navigational joins, along paths formed by properties and/or xpcollections defined in your XPObjects. This applies to XPO as whole, so XPQuery too. All other kinds of joins are called "free joins" and either:

• are silently emulated by XPO by fetching related objects, extracting key values from them and rewriting the query into a multiple roundtrips with a series of partial queries that fetch full objects with WHERE-id-IN-(@p0,@p1,@p2,...) - but this happens only in the some simpliest cases
• or are "not fully supported", meaning they throw exceptions and require you to manually split the query or rephrase it

Possible direct solution schemes

If ITEM_ID is a relation and XPCollection in PRICE class, then you could rewrite your query so that it fetches a PRICE object then builds up a result object and initializes its fields with PRICE object's properties. Something like:

return new ItemPricesViewModel()
{
    Source = (from o in XpoSession.Query<PRICE>().AsEnumerable()
              select new ItemPriceViewModel()
              {
                  ID = o.ITEM_ID.ITEM_ID,
                  ItemCod = o.ITEM_ID.ITEM_COD,
                  ....
                  ItemModifierID = o.ITEM_MODIFIER_ID.ITEM_MODIFIER_ID,
                  ItemPrices = (from d in o
                                where d.ITEM_ID.ITEM_ID == ....
                                select new Price()
    ....          ....          ....
};

Note the 'AsEnumerable' that breaks the query and ensures that PRICE objects are first fetched instead of just trying to translate the query. Very probable that this would "just work".

Also, splitting the query into explicit stages sometimes help the XPO to analyze it:

return new ItemPricesViewModel()
{
    Source = (from o in XpoSession.Query<PRICE>()
              select new
              {
                  id = o.ITEM_ID.ITEM_ID,
                  itemcod = o.ITEM_ID.ITEM_COD,
                  ....
              }
          ).AsEnumerable()
          .Select(temp => 
              select new ItemPriceViewModel()
              {
                  ID = temp.id
                  ItemCod = temp.itemcod,
                  ....
                  ItemPrices = (from d in XpoSession.Query<PRICE>()
                                where d.ITEM_ID.ITEM_ID == ....
                                select new Price()
    ....          ....          ....
};

Here, note that I first fetch the item-data from server, and then conctruct the item on the 'client', and then build the required groupings. Note that I could not refer to the variable o anymore. In these precise case and examples, unsuprisingly, the second one (splitted) would be probably even slower than the first one, since it would fetch all PRICEs and then refetch the groupings through additional queries, while the first one would just fetch all PRICEs and then would calculate the groups in-memory basing on the PRICEs already fetched. This is not an a sideeffect of my laziness, but it is a common pitfall when rewriting the LINQ queries, so I included it as a warning :)

Both of these code examples are NOT RECOMMENDED for your case, as they would probably have very poor performace, especially if you have many PRICEs in the table, which is highly likely. I included them to present as only an example of how you could rewrite the query to siplify its structure so the XPO can eat it without choking. However, you have to be really careful and pay attention to little details, as you can very easily spoil the performance.

observations and real solution

However, it is worth noting that they are not that much worse than your original query. It was itself quite poor, since it tried to perform something near O(N^2) row-fetches from the table just to perform to group te rows by "ITEM_ID" and then formatting the results as separate objects. Properly done, it would be something like O(N lg N)+O(N), so regardless of being supported or not, your alternate attempt with GroupBy is surely a much better approach, and I'm really glad you found it yourself.

Very often when you are trying to split/simplify the XPQuery expressions as I did above, you implicitely rethink the problem and find an easier and simplier way to express the query that was initially not-supported or just were crashing.

Unfortunatelly, your query was in fact quite simple. For a really complex queries that cannot be "just rephrased", splitting into stages and making some of the join-filter work at 'client' is unavoidable.. But again, doing them on XPCollections or XPViews with CritieriaOperators is impossible too, so either we have to bear with it or use plain direct handcrafted SQL..

Sidenote:

Whole XPO has problems with "free joins", they are "not fully supported" not only in XPQuery, but also there's not much for them in XPCollection, XPView, CriteriaOperators, etc, too. But, it is worth noting that at least in "my version" of DX11, the XPQuery has very poor LINQ support at all.

I've hit many cases where a proper LINQ query was:

• throwing "NotSupportedException", mostly in FreeJoins, but also very often with complex GroupBy or Select-with-Projection, GroupJoin, and many others - sometimes even Distinct(!) seemed to malfunction
• throwing "NullReferenceExceptions" at some proper type conversions (XPO tried to interprete a column that held INT/NULL as an object..), often I had to write some completely odd and artificial expressions like foo!=null && foo.bar!=123 instead of foo = 123 despite the 'foo' being an public int Foo {get;set;}, all because the DX could not cope properly with NULLs in the database (because XPO created nullable-INT column for this property.. but that's another story)
• throwing other random ArgumentException/InvalidOperation exceptions from other constructs

• or even analyzing the query structure improperly, for example this one is usually valid:

  session.Query<ABC>()
       .Where( abc => abc.foo == "somefilter" )
       .Select( abc => new { first = abc, b = abc } )
       .ToArray();
  

  but things like this one usually throws:

  session.Query<ABC>()
       .Select( abc => new { first = abc, b = abc } )
       .Where ( temp => temp.first.foo == "somefilter" )
       .ToArray();
  

  but this one is valid:

  session.Query<ABC>()
       .Select( abc => new { first = abc, b = abc } )
       .ToArray()
       .Where ( temp => temp.first.foo == "somefilter" )
       .ToArray();
  

  The middle code example usually throws with an error that reveals that XPO layer were trying to find ".first.foo" path inside the ABC class, which is obviously wrong since at that point the element type isn't ABC anymore but instead a' anonymous class.

disclaimer

I've already noted that, but let me repeat: these observations are related to DX11 and most probably also earlier. I do not know what of that was fixed in DX12 and above (if anything at all was!).

Answer 2

It seems like XPO it's not able to respond to this scenario. For reference this is what you could do with DbContext.

It sounds like maybe you want a GroupBy. Try something like this.

var result = dbContext.Prices
   .GroupBy(p => new {p.ItemName, p.ItemTypeName)
   .Select(g => new Item
                 {
                     ItemName = g.Key.ItemName,
                     ItemTypeName = g.Key.ItemTypeName,
                     Prices = g.Select(p => new Price 
                                                {
                                                    Price = p.Price
                                                }
                                       ).ToList()

                 })
 .Skip(x)
 .Take(y)
 .ToList();