Unit Test Adapter threw exception with no stack trace in TeamCity 7.1.5

Question

We have an ASP.NET MVC 4.5 project. For months it has been compiling and all unit tests passing locally on our development machines and on the TeamCity 7.1.5 machine. Monday we updated to newer versions of some of our dependencies (specifically, to get OData 5.0.0-rc1 for select-expand).

The project compiles fine on our development machines and the build machine. It runs fine on our development machines and on both QA environments it's deployed to. All the unit tests pass on our development machines. However, about half (~300) of the unit tests now fail when run from TeamCity using the MSTest build runner, all with the same, useless error:

Unit Test Adapter threw exception: 
Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.

I've tried everything imaginable. At first I figured there was a legitimate problem. Since there's no way to retrieve the LoaderExceptions property from TeamCity (why doesn't MSTest just display more information!?!?), I logged in to the machine TeamCity is running on, opened up the console, changed into the work directory that our build is running from, and ran the tests manually so that I could retrieve the property. ALL OF THE TESTS PASSED! There goes that theory...

After an entire day of Googling and reading other answers on StackOverflow, I've tried about two dozen different combinations of the following steps:

• Changing the build from "Rebuild" to "Clean," running it without tests so that it just cleans, changing it back, and running it again.
• Changing the build from "Debug" to "Release," running it, and changing it back again.
• Cleaning agent sources from TeamCity admin pages.
• Manually deleting entire work directory off of the agents, skipping the recycle bin and going straight to the netherworlds.
• Deleting all source control (TFS) caches off the build machine and agents.
• Telling TeamCity to clean the work directory before getting latest from source control.

I've reached the end of my troubleshooting capabilities. The tests just won't run anymore on the build machine, for apparently no reason at all.

What can I do? What the heck is going on?

Answer 1

The short answer to this is, "It's a bug."

I reported the bug to JetBrains. It appears that, though there might be some underlying problem with my project, it's only a problem when run from Teamcity, and it's impossible to retrieve the actual error message, so there's no way to know what that problem is. The runner should be catching the exception and printing out the LoaderExceptions property, but it isn't. Instead, it's just calling ToString() on the exception. The runner will have to be updated to look for this specific exception and print the LoaderExceptions property. Once that happens, then we can see the underlying problem, which may result in changing something in the project, or it may result in filing another bug with JetBrains.

Answer 2

Workaround

I spent quite some time debugging our Continuous Integration build failure while experiencing this bug. Since Jetbrains doesn't address the issue, I thought I'd post my workaround here so others have something to work with.

My problem was that the build failed on the step which runs the unit tests. Coincidentally the build which failed added a unit test project. When the newly added unit tests were excluded, everything ran successfully. Because of the reasons stated above and Teamcity not handling the exception properly, debugging options are limited.

I never found a solution which doesn't get your hands dirty, but what you can do is the following. It's basically manually running MSTest versus the test assemblies on the build agent and does require remote desktop access to it. In my case, Teamcity ran the unit tests from the following directory:

C:\BuildAgent\temp\buildTmp[NAME TIMESTAMP]\Out.

Run the following command to initiate the MSTest runner on the assembly which contains failed tests:

C:\BuildAgent\temp\buildTmp\Out>"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe" /testcontainer:"C:\BuildAgent\temp\buildTmp[NAME TIMESTAMP]\Assembly.With.Failed.Tests.dll"

The output should give you the problem which actually makes the tests fail. In my case it turned out that the new project had a reference to Ploeh.AutoFixture with a different version than the version used in the other unit test projects.

Unable to load the test container 'C:\BuildAgent\temp\buildTmp\Out[NAME TIMESTAMP]\Assembly.With.Failed.Tests.dll' or one of its dependencies. Error details: System.IO.FileLoadException: Could not load file or assembly 'Ploeh.AutoFixture, Version=3.18.10.0, Culture=neutral, PublicKeyToken=b24654c590009d4f' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040).

Specific problem solution

In case you're wondering how to fix a similar issue, you can do the following. Open the package manager console (View -> Other windows -> NuGet Package Manager) and execute the following command.

Install-Package Ploeh.AutoFixture -Version 3.18.10.0

This adds the NuGet package (to the project selected on the dropdown) with a specific version, just make sure that all the projects are aligned in this sense.