How to diagnose “TestFixtureSetUp Failed”

Question

We use TeamCity as our CI server, and I've just started seeing "TestFixtureSetUp Failed" in the test failure window.

Any idea how I go about debugging this problem? The tests run fine on my workstation (R# test runner in VS2008).

Answer 1

It is a bit of a flaw in the implementation of TestFixtureSetUp (and TestFixtureTearDown) that any exceptions are not well reported. I wrote the first implementation of them and I never got it to work the way it was supposed to. At the time the concepts in the NUnit code were tightly coupled to the idea that actions were directly related to a single test. So the reporting of everything was related to a test result. There wasn't really a space for reporting something that happened at the suite level without a huge re-write (it isn't a refactoring when you change a sheep into an escalator).

Because of that bit of history it's hard to find out what really happened in a TestFixtureSetUp. There isn't a good place to attach the error. The TestFixtureSetUp call is a side effect of running a test instead of being directly related to it.

@TrueWill has the right idea. Check the logs and then modify the test to add more logging if necessary. You might want to put at try/catch inside the TestFixtureSetup and log a lot in the catch block. I just thought I could add some background to it (in other words it's kind of my fault).

Answer 2

I'd check the Build Log first.

If it's not obvious from that, you could try including Console.WriteLines in the tests - I'm not positive, but I think those are written to the Build Log. Alternately you could log to a file (even using log4net if you wanted to get fancy).

If you have Visual Studio installed on the CI server, you could try running the build/tests from there. If it's a connectivity issue, that might resolve it.

I've seen path issues, though, where relative paths to files were no longer correct or absolute paths were used. These are harder to debug, and might require logging the paths and then checking if they exist on the build server.

Answer 3

I ran into this today when creating some integration tests that have long running setup that I don't want to duplicate. I ended up wrapping all the test fixture setup logic in a try/catch. I then add a SetUp method whose sole purpose is to see if a failure occurred during fixture setup and provide better logging.

Exception testFixtureSetupException = null;

[TestFixtureSetUp]
public void FixtureSetup()
{
    try
    {
        // DoTestFixtureSetup
    }
    catch (Exception ex)
    {
        testFixtureSetupException = ex;
    }
}

[SetUp]
// NUnit doesn't support very useful logging of failures from a TestFixtureSetUp method. We'll do the logging here.
public void CheckForTestFixturefailure()
{         
    if (testFixtureSetupException != null)
    {
        string msg = string.Format("There was a failure during test fixture setup, resulting in a {1} exception. You should check the state of the storage accounts in Azure before re-running the RenewStorageAccountE2ETests. {0}Exception Message: {3}{0}Stack Trace:{4}",
            Environment.NewLine, testFixtureSetupException.GetType(), accountNamePrefix, testFixtureSetupException.Message, testFixtureSetupException.StackTrace);
        Assert.Fail(msg);
    }
 }

Answer 4

I was getting the same error while running any test with SpecFlow using Visual NUnit. When I tried doing the same from the Unit Test Explorer(provided by Resharper), it gave a little more helpful message: Binding methods with more than 10 parameters are not supported. I realized I can't have a SpecFlow method with more than 10 params, had to remove the test.

Answer 5

I was able to see that I was not creating my test database correctly by doing a quick switch to VS Unit Testing. In my Case it was able to return a better response to the reason why it failed. I usually use NUnit. "Unable to create instance of class X. Error: System.Data.SqlClient.SqlException: A file activation error occurred. The physical file name '\DbTest.mdf' may be incorrect. Diagnose and correct additional errors, and retry the operation. CREATE DATABASE failed. Some file names listed could not be created. Check related errors.. "

Answer 6

Run the unit test in debug mode. You may find a runtime error in the the setup.

Answer 7

If you are using SpecFlow and C# in Visual Studio, look at the auto-generated <whatever>.feature.cs file after the test fails. On the public partial class <whatever>Feature line, you should see a symbol which when hovered over will show the reason that the NUnit fixture setup failed. In my case, it was that some of my BeforeFeature methods in my TestHooks class were not static. All BeforeTestRun, AfterTestRun, BeforeFeature, and AfterFeature methods need to be static.

Answer 8

I had this issue and it was caused by adding a private readonly Dictionary in the class, much the same way that you add a private const string.

I tried to make the Dictionary a constant but you can't do that at compile time. I solved this by putting my Dictionary in a method that returns it.

Answer 9

I had this symptom caused by an error during field initialization. If your initialize your fields in the [SetUp] method, you should see a better error message.

[TestFixture]
internal class CommandParserTest
{
    // obscure error message
    private CommandParser parser = new CommandParser(...);
    ...
}

[TestFixture]
internal class CommandParserTest
{
    private CommandParser parser;

    [SetUp]
    public void BeforeTest()
    {
        // better error message
        parser = new CommandParser(...);
    }
    ...
}

Answer 10

I was troubled by this today. I did the following to get the actual error.

(1) Write another test in a separate fixture which initializes an instance of the troubling test fixture, explicitly calls setup methods such as TestFixtureSetUp and SetUp if any, and then executes the target test method.

(2) Add exception handling code for the new code above, and log / output the actual exception to somewhere.

Answer 11

In case it can help someone: You can catch the exception and write it in the console on the TearDown

Something like :

[SetUpFixture]
public class BaseTest
{
    private Exception caughtException = null;

    [SetUp]
    public void RunBeforeAnyTests()
    {
        try
        {
            throw new Exception("On purpose");
        }
        catch (Exception ex)
        {
            caughtException = ex;               
        }
    }

    [TearDown]
    public void RunAfterAnyTests()
    {
        if (caughtException != null)
        {
            Console.WriteLine(string.Format("TestFixtureSetUp failed in {0} - {1}", this.GetType(), caughtException.Message));
        }           
    }

}

And the result will be :

TestFixtureSetUp failed in IntegratedTests.Services.BaseTest - On purpose