Pex raises NullReferenceException when exploring

Question

I'm writing a server component in C# and use Pex for unittesting.

I've a complex parameterized unit test for a specific method. Now it turns out that as soon as i add a certain assert block, some pex exploration runs come up as failed with a NullReferenceException on the closing line of my method (right at the bracket). When i debug a failed run it runs absolutely fine.

Did i make a mistake or is this a bug in pex?

Thanks!

[PexMethod]
public Task Start(CancellationToken cancellationToken,
    int workingEndpoints, // endpoints that run succesfully
    int failingEndpoints, // endpoints that fail immidiatly
    int brokenEndpoints) // endpoints that return null for their task
{
    PexAssume.IsTrue(workingEndpoints >= 0);
    PexAssume.IsTrue(failingEndpoints >= 0);
    PexAssume.IsTrue(brokenEndpoints >= 0);
    PexAssume.IsTrue(workingEndpoints + failingEndpoints + brokenEndpoints >= 1);

    // create fake endpoints based on the count
    List<IHostEndpoint> fakeEndpoints = new List<IHostEndpoint>();
    Exception failedTaskException = new Exception();
    // Create a few endpoint stubs for testing purposes and add them to the  list (commented away for relevance)

    // create and start the host
    Host host = new Host(fakeEndpoints.ToArray());
    Task result = host.Start(cancellationToken);

    PexAssert.IsNotNull(result);
    if (failingEndpoints > 0 || brokenEndpoints > 0)
    {
        PexAssert.IsNotNull(result.Exception);

        int failedEndpointExceptionCount = 0;
        int brokenEndpointExceptionCount = 0;

        result.Exception.Flatten().Handle(innerException =>
        {
            if (innerException == failedTaskException)
                failedEndpointExceptionCount++;
            else
                brokenEndpointExceptionCount++;

            return true;
        });

        // after one broken endpoint, the run method should stop starting more endpoints
        int brokenEndpointExpectedCount = Math.Min(1, brokenEndpoints);
        PexAssert.AreEqual(failedEndpointExceptionCount, failingEndpoints);
        PexAssert.AreEqual(brokenEndpointExceptionCount, brokenEndpointExpectedCount); 
    }

    return result;            
}

EDIT

One assumption might be that due to async code, Pex encounters some problems. I've checked every single run and even faked the start method of host. There are no async methods. I do create 1 task in some cases but i run it synchronously (proof below)

Task endpointTask = endpoint.Start(innerCancellationToken);                

if (endpointTask == null)
{
    // This endpoint is broken, for simplicity we raise an exception in the normal pipe
    Task faultedTask = new Task(() =>
    {
        throw new InvalidOperationException("Endpoint returned a null valued task which is not allowed");
    });

    faultedTask.RunSynchronously();
    innerTasks.Add(faultedTask);

    break;
}
else
{
    innerTasks.Add(endpointTask);
}

The IHostEndpoint stubs are created using a TaskCompletionSource with a value/state directly set.

Answer 1

Pex is a fantastic tool, but it has bugs. From your statements I can tell that this is a bug in Pex: Adding an assert should not cause an unrelated NullRef. I suggest you report this in the Pex forums.