Ncrunch all test pass first run, but fails after code change and when run all button is pressed

Question

I'm running ncrunch, in a new MVC 4 solution in VS2012 using nunit and ninject.

When I first open the solution all 50 or so test run and pass successfully.

After I make any code change (even just a added empty space) ncrunch reports that most of my unit test fail. The same thing happens if I press the 'run all tests' in the ncrunch window.

But if you hit the 'Run all tests visible here' button all 50 test pass again and ncrunch reports everything is fine.

Also when you run each test individually they pass every time.

When they do fail they seem to be failing in my ninject setup code

Error: TestFixtureSetUp failed in ControllerTestSetup

public class ControllerTestSetup
{

    [SetUp]
    public void InitIntegrationTest()
    {
        var context = IntegrationTestContext.Instance;
        context.Init();
        context.NinjectKernel.Load<MediGapWebTestModule>();
    }

    [TearDown]
    public void DisposeIntegrationTest()
    {
        IntegrationTestContext.Instance.Dispose();
    }
}

public class IntegrationTestContext : IDisposable
{  

    private static IntegrationTestContext _instance = null;
    private static readonly object _monitor = new object();

    private IntegrationTestContext() { }

    public static IntegrationTestContext Instance
    {
        get
        {
            if (_instance == null)
            {
                lock (_monitor)
                {
                    if (_instance == null)
                    {
                        _instance = new IntegrationTestContext();
                    }
                }
            }

            return _instance;
        }
    }
}

All the test also run in the resharper test runner without problems every time.

Does anyone know what could be causing this?

I guessing its something to do with the singleton lock code inside the Instance property but I'm not sure.

============================================================================== Progress:

I was able to track this down to a error in the ninject setup method above by wrapping it in a try catch statement, and writing the error to the output window.

The exception was caused by trying to load a module more than once, even tho I definitely haven't and I don't use any type of automatic module loading.

This happen on the lines

LocalSessionFactoryModule.SetMappingAssemblies(() => new[] { typeof(ProviderMap).Assembly });

_kernel.Load<LocalSessionFactoryModule>();
_sessionFactory = _kernel.Get<ISessionFactory>();

where LocalSessionFactoryModule is the ninject module class derived for NinjectModule class.

Why is this only happening with ncrunch and what can I do to solve this issue? Is there a way to check if a module has already been loaded?

Answer 1

NCrunch will never execute tests concurrency within the same process, so unless you have multi-threaded behaviour inside your test logic, then it should be safe to say that this isn't an issue caused by the locking or threading over the singleton.

As you've already tried disabling parallel execution and this hasn't made a difference, I'm assuming that the problem wouldn't be caused by concurrent use of resources outside the test runner process (i.e. files on disk).

This means that the problem is almost certainly related to the sequence in which the tests are being executed. Almost all manual test runners (including Resharper) will run tests in a defined sequence from start to finish. This is good for consistency, but it can mask problems that may surface when the tests are run in an inconsistent/random order. NCrunch will execute tests in order of priority and can also reuse test processes between test runs, which can make the runtime behaviour of your tests different if they haven't been designed with this in mind.

A useful way to surface (and thus debug) sequence related issues is to try running your tests in a manually defined order by using NCrunch. If you right-click a test inside the NCrunch Tests Window, under the 'Advanced' menu you'll find an option to run a test using an existing task runner process. Try this action against several of your tests to see if you can reproduce the sequence that surfaces the problem. When it happens, you should easily be able to get a debugger onto the test and find out why it is failing.

Most sequence related problems are caused by uncleared static members, so make sure each of your tests is written in the assumption that existing state may be left behind by another test that has been run within the process. Another option is to ensure all state is fully cleared by tests on tear down (although in my opinion, this is often a less pragmatic approach).