Unit Test Assert against end result or verifying whether the parameters were called using Moq

Question

Below is a class (Class1) that I want to test, but I'm not fully satisfied with my Unit Test. Please see below code samples.

System Under Test

public interface IRepository {
    string GetParameter(int id);
}

public class Repository {
    public string GetParameter(int id) {
        return "foo";
    }
}

public class ErrorInfo {
    public string ErrorCodes { get; set; }
}

public interface IErrorProvider {
    ErrorInfo BuildErrorMessage(string errorCodes);
}

public class ErrorProvider {
    public ErrorInfo BuildErrorMessage(string errorCodes) {
        return new ErrorInfo(){ErrorCodes = errorCodes};
    }
}

public class Class1 {
    private readonly IRepository _repository;
    private readonly IErrorProvider _errorProvider;
    public Class1(IRepository repository, IErrorProvider errorProvider) {
        _repository = repository;
        _errorProvider = errorProvider;
    }

    public List<ErrorInfo> GetErrorList(int id) {
        var errorList = new List<ErrorInfo>();
        string paramName = _repository.GetParameter(id);

        if (string.IsNullOrEmpty(paramName)) {
            string errorCodes = string.Format("{0}, {1}", 200, 201);
            var error = _errorProvider.BuildErrorMessage(errorCodes);
            errorList.Add(error);
        }

        return errorList;
    }
}

Unit Tests

Below test passes and we check whether the correct error codes being used within the system under test.

[TestMethod]
public void GetErrorList_WhenParameterIsEmpty_ReturnsExpectedErrorCodes2() {
        //Arrange
        var stubRepo = new Mock<IRepository>();
        stubRepo.Setup(x => x.GetParameter(It.IsAny<int>())).Returns(string.Empty);

        var stubErrorMock = new Mock<IErrorProvider>();

        const int id = 5;
        var sut = new Class1(stubRepo.Object, stubErrorMock.Object);

        //Act
        var result = sut.GetErrorList(id);

        //Verify
        string verifiableErrorCodes = "200, 201";
        stubErrorMock.Verify(x => x.BuildErrorMessage(verifiableErrorCodes));
}

However I would prefer testing the end result. For example, I want to Assert against the error codes that have been used in the production code. Below test fails but I like to know your thoughts on how to Assert against the errorCodes that has been used in the system under test.

[TestMethod]
public void GetErrorList_WhenParameterIsEmpty_ReturnsExpectedErrorCodes1() {
        //Arrange
        var stubRepo = new Mock<IRepository>();
        stubRepo.Setup(x => x.GetParameter(It.IsAny<int>())).Returns(string.Empty);

        string expectedErrorCodes = "200, 201";
        var stubErrorRepo = new Mock<IErrorProvider>();
        stubErrorRepo.Setup(e => e.BuildErrorMessage(It.IsAny<string>()));

        const int id = 5;
        var sut = new Class1(stubRepo.Object, stubErrorRepo.Object);

        //Act
        var result = sut.GetErrorList(id);

        //Assert
        Assert.AreEqual(expectedErrorCodes, result.Single().ErrorCodes);
}

What would be the correct way to test this error codes that has been used in the system?

Answer 1

There is not really right or wrong answer and we have decided to use the Assert test as it test the end result.

I took the TDD approach and re-implemented/analysed as below.

• Start with a failing test (to simplify the code I removed the Repository from both test and the sut)

//AssertTest

    [TestMethod]
    public void GetErrorList_WhenParameterIsEmpty_ReturnsExpectedErrorCodes1()
    {
        //Arrange
        const string expectedErrorCodes = "200, 201";
        var stubErrorRepo = new Mock<IErrorProvider>();
        stubErrorRepo.Setup(e => e.BuildErrorMessage(expectedErrorCodes)).Returns(new ErrorInfo() { ErrorCodes = expectedErrorCodes });

        var sut = new Class1(stubErrorRepo.Object);

        //Act
        var result = sut.GetErrorList();

        //Assert
        Assert.AreEqual(expectedErrorCodes, result.Single().ErrorCodes);
    }

//SUT

    public IEnumerable<ErrorInfo> GetErrorList(int id)
    {           
        yield return new ErrorInfo();            
    }

As you would expect the test fail.

Now if write enough production code to make this test pass.

    public IEnumerable<ErrorInfo> GetErrorList()
    {
        yield return _errorProvider.BuildErrorMessage("200, 201");
    }

The VerifyTest would still fail for the above SUT.

//VerifyTest

   [TestMethod]
    public void GetErrorList_WhenParameterIsEmpty_ReturnsExpectedErrorCodes2()
    {
        //Arrange
        var stubErrorMock = new Mock<IErrorProvider>();
        var sut = new Class1(stubErrorMock.Object);

        //Act
        sut.GetErrorList();

        //Verify
        string verifiableErrorCodes = "200, 201";
        stubErrorMock.Verify(x => x.BuildErrorMessage(verifiableErrorCodes));
    }

However if I want this test to pass, I can write the below production code as below

    public IEnumerable<ErrorInfo> GetErrorList()
    {
        _errorProvider.BuildErrorMessage("200, 201");
        return null;
    }

Now the VerifyTest passes, but the AssertTest fails.

Both tests are valid in their own ways. However they test different semantics. AssertTest test whether the end result contains the correct error codes. Verify test ensures the method is called with the correct error codes. It is important to note that the end value of the Assert test is based on the setup method "match" provided by the Moq framework. In other words the setup dictates what the end result would be. AssertTest would fail if the setup is configured incorrectly or the production code uses error codes that does not match the setup configuration.

It is preferred to use the AssertTest as it test the end result.

Answer 2

I suggest to mock only the IRepository and use a real IErrorProvider. Then you can call GetErrorList(id) and check the result.