How to get PEX to automatically generate inputs for code involving LINQ

Question

I'm having trouble getting PEX to automatically cover methods calling Linq extension methods, such as Where() and Contains() in this example:

public class MyEntity
{
    public int Id { get; set; }
}

public interface IWithQueryable
{
    IQueryable<MyEntity> QueryableSet();
}

public class ConsumerOfIhaveIQueryable
{
    private readonly IWithQueryable _withIQueryable;
    public ConsumerOfIhaveIQueryable(IWithQueryable withIQueryable)
    {
        // <pex>
        Contract.Requires<ArgumentNullException>(
            withIQueryable != null, "withIQueryable");
        // </pex>
        _withIQueryable = withIQueryable;
    }

    public IEnumerable<MyEntity> GetEntitiesByIds(IEnumerable<int> ids)
    {
        Contract.Requires<ArgumentNullException>(ids != null, "ids");
        // <pex>
        Contract.Assert
            (this._withIQueryable.QueryableSet() != (IQueryable<MyEntity>)null);
        // </pex>
        IEnumerable<MyEntity> entities =
        _withIQueryable.QueryableSet().Where(
            entity => ids.Contains(entity.Id));
        if (entities.Count() != ids.Count())
        {
            return null;
        }
        return entities;
    }
}

[PexClass(typeof(ConsumerOfIhaveIQueryable))]
[PexAllowedExceptionFromTypeUnderTest(typeof(InvalidOperationException))]
[PexAllowedExceptionFromTypeUnderTest(typeof(ArgumentException), AcceptExceptionSubtypes = true)]
[TestClass]
public partial class ConsumerOfIhaveIQueryableTest
{
    [PexMethod]
    public IEnumerable<MyEntity> GetEntitiesByIds(
        [PexAssumeUnderTest]ConsumerOfIhaveIQueryable target,
        int[] ids)
    {
        var result = target.GetEntitiesByIds(ids);
        PexAssert.IsTrue(result.Count() == ids.Length);
        return result;
    }
}

When I'm running PEX exploration on this PexMethod I'm seeing the following problems:

• I keep getting the same exception and PEX keeps suggesting the same "invariant" fix in the form of the Contract.Assert that you see in the // region: I believe the problem is somehow related to how Pex relates to Linq but I'm not sure

--- Description failing test: ArgumentNullException, Value cannot be null. Parameter name: source

[TestMethod]
[PexGeneratedBy(typeof(ConsumerOfIhaveIQueryableTest))]
[PexRaisedException(typeof(ArgumentNullException))]
public void GetEntitiesByIdsThrowsArgumentNullException385()
{
    using (PexChooseBehavedBehavior.Setup())
    {
      SIWithQueryable sIWithQueryable;
      ConsumerOfIhaveIQueryable consumerOfIhaveIQueryable;
      IEnumerable<MyEntity> iEnumerable;
      sIWithQueryable = new SIWithQueryable();
      consumerOfIhaveIQueryable =
        ConsumerOfIhaveIQueryableFactory.Create((IWithQueryable)sIWithQueryable);
      int[] ints = new int[0];
      iEnumerable = this.GetEntitiesByIds(consumerOfIhaveIQueryable, ints);
    }
}

--- Exception details

System.ArgumentNullException: Value cannot be null. Parameter name: source at System.Linq.IQueryable'1 System.Linq.Queryable.Where(System.Linq.IQueryable'1 source, System.Linq.Expressions.Expression'1> predicate) c:\users\moran\documents\visual studio 2010\Projects\PexTuts\PexIQueryable\PexIQueryable\ConsumerOfIhaveIQueryable.cs(29): at System.Collections.Generic.IEnumerable'1 PexIQueryable.ConsumerOfIhaveIQueryable.GetEntitiesByIds(System.Collections.Generic.IEnumerable`1 ids) c:\users\moran\documents\visual studio 2010\Projects\PexTuts\PexIQueryable\PexIQueryable.Tests\ConsumerOfIhaveIQueryableTest.cs(34): at System.Collections.Generic.IEnumerable'1 PexIQueryable.ConsumerOfIhaveIQueryableTest.GetEntitiesByIds(PexIQueryable.ConsumerOfIhaveIQueryable target, System.Int32[] ids)

• I can't get PEX to generate relevant inputs. As you can see, I tried to "help" it by adding a PexAssert and a branch in my code, but this branch is never covered, even though it should be relatively simple to generate a code that would walk that path. PEX only tries to pass null or an empty array as the list of Ids (I read somewhere that it is easier for PEX to work with arrays (int[]) rather than IEnumerable.

Would love to get some comments on this...

BTW, this is my first SO post, Hope I didn't spam with too much code and info.

Moran

Answer 1

After some time setting up the code for this, I've made a few assumptions. I'm assuming that you're stubbing the IWithQueryable via a Moles stub and also that the NullArgumentException occurs when you remove the Contract assertion that the QueryableSet() method doesn't return null.

As for the code, IMO the more code the better, as long as it's relevant- far better to have too much than too little to go on, so that's fine. As above, do try to make clear all the assumptions in the code (e.g. the Moles stubbing (as there's different ways to achieve this and it's something one has to assume).

I'm not 100% sure what you're asking. The code is failing because the stubbed IWithQueryable object doesn't have an implmementation for the QueryableSet() method and that method returns null. The PexAssert here won't help it figure out how to create a LINQ provider, which is what you're asking it to do. The PexChooseBehavedBehavior.Setup() simply replaces any calls to delegates on the Moles stubs (which don't have a custom delegate) with the default behaviour which is default(T), so that's why the source is null- the QueryableSet() is initialised to null.

You can solve this in a few ways (at least in the sense of providing a way of creating the QueryableSet() method). You can create a factory method to generate either the whole SIWithQueryable, or just the QueryableSet delegate. This is something that Pex suggests (however, with me it got the types and namespaces muddled-up). For instance:

/// <summary>A factory for Microsoft.Moles.Framework.MolesDelegates+Func`1[System.Linq.IQueryable`1[StackOverflow.Q9968801.MyEntity]] instances</summary>
public static partial class MolesDelegatesFactory
{
    /// <summary>A factory for Microsoft.Moles.Framework.MolesDelegates+Func`1[System.Linq.IQueryable`1[StackOverflow.Q9968801.MyEntity]] instances</summary>
    [PexFactoryMethod(typeof(MolesDelegates.Func<IQueryable<MyEntity>>))]
    public static MolesDelegates.Func<IQueryable<MyEntity>> CreateFunc()
    {
        throw new InvalidOperationException();

        // TODO: Edit factory method of Func`1<IQueryable`1<MyEntity>>
        // This method should be able to configure the object in all possible ways.
        // Add as many parameters as needed,
        // and assign their values to each field by using the API.
    }

    /// <summary>A factory for Microsoft.Moles.Framework.MolesDelegates+Func`1[System.Linq.IQueryable`1[StackOverflow.Q9968801.MyEntity]] instances</summary>
    [PexFactoryMethod(typeof(SIWithQueryable))]
    public static SIWithQueryable Create()
    {
        var siWithQueryable = new SIWithQueryable();
        siWithQueryable.QueryableSet = () => { throw new InvalidOperationException(); };

        return siWithQueryable;
        // TODO: Edit factory method of Func`1<IQueryable`1<MyEntity>>
        // This method should be able to configure the object in all possible ways.
        // Add as many parameters as needed,
        // and assign their values to each field by using the API.
    }
}

and then hook it up to the test method with one of the two lines assigning sIWithQueryable:

[TestMethod]
[PexGeneratedBy(typeof(ConsumerOfIhaveIQueryableTest))]
public void GetEntitiesByIdsThrowsArgumentNullException678()
{
  SIWithQueryable sIWithQueryable;

  // Either this for the whole object.
  sIWithQueryable = MolesDelegatesFactory.Create();

  // Or this for just that delegate.
  sIWithQueryable = new SIWithQueryable();
  sIWithQueryable.QueryableSet = MolesDelegatesFactory.CreateFunc();

  ConsumerOfIhaveIQueryable consumerOfIhaveIQueryable;
  IEnumerable<MyEntity> iEnumerable;
  consumerOfIhaveIQueryable = ConsumerOfIhaveIQueryableFactory.Create((IWithQueryable) sIWithQueryable);
  int[] ints = new int[0];
  iEnumerable = this.GetEntitiesByIds(consumerOfIhaveIQueryable, ints);
}

This will then call your factory methods when creating the stub for IWithQueryable. This is still a problem, as regenerating the explorations will wipe out the stub setup.

If you provide the parameterless factory method to create the stub (MolesDelegatesFactory.CreateFunc()), then Pex will know about this and generate the tests to use it. So, it will correctly manage the behaviour across test regenerations. Unfortuantely, Pex suggests creating this delegate as a factory method- however, it is never called, the default implementation is always used, it seems one has to mock the parent type.

However, I'm wondering why you're creating an interface IWithQueryable that simple wraps another, and also, what you expect to do with the IQueryable. In order to do anything very useful, you'll have a lot of work going on to deal with the IQueryable interface - the Provider and Expression mainly, you'll pretty-much have to write a mock query provider, which will not be easy.