LINQ query to exclude overridden values?

Question

I have some tables as follows:

ImageSettingOverrides

TechniqueSettings

SettingKeyValues

From TechniqueSettings table:
BAZ-FOO setting (SettingKeyId: 7) is an override to the BAZ-Default (SettingKeyId: 4) setting.

Example of expected return from query grouped by Override value:

I want to compile a list of SettingKeyValues given technique BAZ and override FOO that excludes the overridden BAZ-Default settings and includes non-overridden BAZ-Default settings.

I currently have a LINQ query that groups setting-key values based on Default/Override values:

var techniqueSettings = _dataRepository.All<TechniqueSetting>()
   .Where(s => s.Technique.Equals(TechniqueName, StringComparison.InvariantCultureIgnoreCase))
   // group settings by: e.g. Default | FOO
   .GroupBy(s => s.Override);

From there I determine if the user is querying for just the defaults or the defaults with overrides:

 var techniqueGroups = techniqueSettings.ToArray();
 if (OverridesName.Equals("Default", StringComparison.InvariantCultureIgnoreCase)) {
    // get the default group and return as queryable
    techniqueSettings = techniqueGroups
       .Where(grp => grp.Key.Equals("Default", StringComparison.InvariantCultureIgnoreCase))
       .AsQueryable();
 } else {
    // get the overrides group - IGrouping<string, TechniqueSetting>
    var overridesGroup = techniqueGroups
       .Where(grp => !grp.Key.Equals("Default", StringComparison.InvariantCultureIgnoreCase))
       .First();

    var defaultGroup = techniqueGroups
       .Where(grp => grp.Key.Equals("Default", StringComparison.InvariantCultureIgnoreCase))
       // we know what is in the overrides, so exlude them from being selected here
       // how to exlude overridden defaults???
       .First();
 }

In addition, I can't help but think there must be an easier - less clumsy - LINQ query using JOIN (maybe ???).

NOTE: Using EntityFramework 6.x
__
UPDATE: I found Aggregate seems to simplify somewhat but still required an anonymous method.

var defaultGroup = techniqueGroups
   .Where(grp => grp.Key.Equals("Default", StringComparison.InvariantCultureIgnoreCase))
   .Aggregate(overridesGroup,
      (overrides, defaults) => {
         var settings = new List<TechniqueSetting>();
         foreach (var setting in defaults) {
            if (overrides.Any(o => o.SettingKey.Key == setting.SettingKey.Key)) {
               continue;
            }

            settings.Add(setting);

         }

         return settings.GroupBy(s => s.Override).First();
      },
      setting => setting);

I haven't tried the Join yet per comment by @MarkoDevcic.

Can Except be used in this query?

Answer 1

I found Aggregate seems to simplify somewhat but still required an anonymous method.

var defaultGroup = techniqueGroups
   .Where(grp => grp.Key.Equals("Default", StringComparison.InvariantCultureIgnoreCase))
   .Aggregate(overridesGroup,
      (overrides, defaults) => {
         var settings = new List<TechniqueSetting>();
         foreach (var setting in defaults) {
            if (overrides.Any(o => o.SettingKey.Key == setting.SettingKey.Key)) {
               continue;
            }

            settings.Add(setting);

         }

         return settings.GroupBy(s => s.Override).First();
      },
      setting => setting);

Update:

I came up with a couple of extension methods that allows for exclusion of items and comparisons and replacements:

  internal static IEnumerable<TSource> Exclude<TSource>(this IEnumerable<TSource> Source, Func<TSource, bool> Selector) {
     foreach (var item in Source) {
        if (!Selector(item)) {
           yield return item;
        }
     }
  }  

  internal static IEnumerable<TResult> ReplaceWith<TSource1, TSource2, TResult>(this IEnumerable<TSource1> Source1,
     Func<TSource1, TResult> Source1Result,
     IEnumerable<TSource2> Source2,
     Func<TSource1, IEnumerable<TSource2>, TResult> Selector) {

     foreach (var item in Source1) {
        var replaceWith = Selector(item, Source2);

        if (replaceWith == null) {
           yield return Source1Result(item);
           continue;
        }

        yield return replaceWith;
     }
  }

Exclude is fairly straightforward. For ReplaceWith usage:

     var settings = _repository.Settings
        .ReplaceWith(s => s.SettingKeyValue,
           _repository.SettingOverrides.Where(o => o.OverrideName == overrideName),
           (s, overrides) => overrides.Where(o => o.Setting == s)
              .Select(o => o.SettingKeyValueOverride)
              .FirstOrDefault())
        .ToList();

Answer 2

Revised Answer

With values

int myImageId = 1;
string myOverride = "FOO";
string myTechnique = "BAZ";

results =
ImageId Override Value
1 FOO 1000

With Values

int myImageId = 1;
string myOverride = "Default";
string myTechnique = "BAZ";

results =
ImageId Override Value
1 Default 10000

    void Main()
    {

        // Create Tables and Initialize values
        // ***********************************
        var techniqueSettings = new List<TechniqueSettings>();
        techniqueSettings.Add(new TechniqueSettings { Id = 1, Override = "Default", SettingKeyId = 3, Technique="BAZ"});
        techniqueSettings.Add(new TechniqueSettings { Id = 2, Override = "Default", SettingKeyId = 4, Technique="BAZ"});
        techniqueSettings.Add(new TechniqueSettings { Id = 3, Override = "FOO", SettingKeyId = 7, Technique="BAZ"});
        techniqueSettings.Add(new TechniqueSettings { Id = 4, Override = "FOO", SettingKeyId = 8, Technique="BAZ"});    

        var imageSettingOverrides = new List<ImageSettingOverrides>();
        imageSettingOverrides.Add(new ImageSettingOverrides {SettingId = 1, ImageId=1, Override=null } );
        imageSettingOverrides.Add(new ImageSettingOverrides {SettingId = 2, ImageId=1, Override="FOO" } );
        imageSettingOverrides.Add(new ImageSettingOverrides {SettingId = 3, ImageId=1, Override="FOO" } );

        var settingKeyValues = new List<SettingKeyValues>();
        settingKeyValues.Add(new SettingKeyValues {Id = 4, Setting="Wait", Value=1000 } );
        settingKeyValues.Add(new SettingKeyValues {Id = 7, Setting="Wait", Value=10000 } );


        int myImageId = 1;
        string myOverride = "FOO";
        string myTechnique = "BAZ";

        var results = from iso in imageSettingOverrides
                join ts in techniqueSettings on iso.SettingId equals ts.Id
                join skv in settingKeyValues on ts.SettingKeyId equals skv.Id
                where iso.ImageId == myImageId &&
                    //iso.Override.Equals(myOverride,StringComparison.InvariantCultureIgnoreCase)  &&
                    ts.Override.Equals(myOverride,StringComparison.InvariantCultureIgnoreCase)  &&
                    ts.Technique.Equals(myTechnique, StringComparison.InvariantCultureIgnoreCase) 
                select new {
                    ImageId = iso.ImageId,
                    Override = ts.Override,
                    Value = skv.Value
                };


        results.Dump();
    }

    // Define other methods and classes here
    public class ImageSettingOverrides
    {
        public int SettingId {get; set;}
        public int ImageId {get; set;}
        public string Override {get; set;}
    }

    public class TechniqueSettings
    {
        public int Id {get; set;}
        public string Override {get; set;}
        public int SettingKeyId {get; set;}
        public string Technique { get; set;}
    }

    public class SettingKeyValues
    {
        public int Id {get; set;}
        public String Setting {get; set;}
        public int Value {get; set;}
    }

Answer 3

I assume you expect each of SettingKeyValues in the result will have unique Setting value (it doesn't make sense to have two 'Wait' records with different numbers against them).

Here is query:

        var result =
        (
            from ts in techniqueSettings
            // For only selected technique
            where ts.Technique.Equals("BAZ", StringComparison.InvariantCultureIgnoreCase)

            // Join with SettingsKeyValues
            join skv in settingKeyValues on ts.SettingKeyId equals skv.Id

            // intermediate object
            let item = new { ts, skv }

            // Group by SettingKeyValues.Setting to have only one 'Wait' in output
            group item by item.skv.Setting into itemGroup

            // Order items inside each group accordingly to Override - non-Default take precedence
            let firstSetting = itemGroup.OrderBy(i => i.ts.Override.Equals("Default") ? 1 : 0).First()

            // Return only SettingKeyValue
            select firstSetting.skv
         )
         .ToList();   

Answer 4

I'm going to make some assumptions.

1. That if there is an ImageSettingOverrides the override also has to match the override passed in AKA FOO (that's the where iSettingsOverrides => iSettingsOverrides.Override == OverridesName in the join clause
2. You only want a distinct list of SettingKeyValues
3. TechniqueSetting.Id is the key and ImageSettingOverride.TechniqueSettingsId is the foreign key and that's how they are related
4. SettingKeyValue.Id is the key and TechniqueSetting.SettingKeyId is the foreign key and that's how they are related.
5. You don't have navigation properties and I have to do the join.

If I understand your classes and how they are related this should give you a list of SettingKeyValues. Since everything stays IQueryable it should execute on the server.

//I'm assuming these are your variables for each IQueryable
IQueryable<TechniqueSetting> techniqueSettings;
IQueryable<ImageSettingOverride> imageSettingOverrides;
IQueryable<SettingKeyValue> settingKeyValues;

var OverridesName = "FOO";
var TechniqueName = "BAZ";

IQueryable<TechniqueSetting> tSettings;
if (OverridesName.Equals("Default", StringComparison.InvariantCultureIgnoreCase))
{
    // Get a list of TechniqueSettings that have this name and are default
    tSettings = techniqueSettings.Where(t => t.Override == OverridesName && t.Technique == TechniqueName);
}
else
{
    // Get a list of TechniqueSettings Id that are overridden 
    //  The ImageSettingOverrides have the same override 
    var overriddenIDs = techniqueSettings.Where(t => t.Technique == TechniqueName && t.Override == "Default")
                                         .Join(
                                             imageSettingOverrides.Where(
                                                 iSettingsOverrides =>
                                                 iSettingsOverrides.Override == OverridesName),
                                             tSetting => tSetting.SettingKeyId,
                                             iSettings => iSettings.TechniqueSettingsId,
                                             (tSetting, iSettingsOverrides) => tSetting.Id);

    // Get a list of techniqueSettings that match the override and TechniqueName but are not part of the overriden IDs
    tSettings =
        techniqueSettings.Where(
            t =>
            t.Technique == TechniqueName && !overriddenIDs.Contains(t.Id) &&
            (t.Override == OverridesName || t.Override == "Default"));
}

// From expected results seems you just want techniqueSettings and that's what would be in techniqueSettings right now.
// If you want a list of SettingKeyValues (which is what is stated in the questions we just need to join them in now)
var settings = tSettings.Join(settingKeyValues, tSetting => tSetting.SettingKeyId,
                              sKeyValues => sKeyValues.Id, (tSetting, sKeyValues) => sKeyValues)
                        .Distinct();