Microsoft Fakes & TFS 2012 Visual Studio Test Runner - unit test fail sporadically

Question

Environment

• Visual Studio 2012 Premium Update 3
• Team Foundation Server 2012 Update 3
• edit: .NET Framework 4
• DefaultTemplate (DefaultTemplate.11.1.xaml)
  • Visual Studio Test Runner
    
    • Target Platform: X86

Problem

I have 2 Unit-Test-Assemblies:

• ExtensionTests
• UserTests

both make use of the Microsoft Fakes Framework (using Stubs and Shims).

Running the unit tests locally works fine (tested on 4 different machines, even on the Visual Studio installed on the build server), however if we build using a build agent, some unit test fails with the exception, e.g.:

Unable to create instance of class UserTests.ClientUserTest. Error: System.TypeLoadException: Could not load type 'WorldDirect.CCM.Shared.Backend.SmartClassic.Fakes.StubClient' from assembly 'WorldDirect.Smart.Backend.Fakes, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.

or the same for a shim:

Test method ExtensionTests.ExtensionTests.UpdateExtensionValidate_NoGrnp_ChecksIpPbxDependencies threw exception: System.TypeLoadExceptio: Could not load type                'WorldDirect.CCM.Shared.Backend.SmartClassic.Fakes.ShimIpPbxRemoteDestination' from assembly 'WorldDirect.Smart.Backend.Fakes, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.

The interesting thing is, they fail sporadically, I've tried to make up a little statistics:

 #     | Outcome          | Comment  
 19    |  64/64 passed    |  only 'UserTests'-UnitTests run, 'ExtensionTests' inactive  
 20    |  37/64 passed    |  same source code as #19  
 21    |  64/64 passed    |  same source code as #19  

 22    |  64/66 passed    |  all 'UserTests' run as well as 2 'ExtensionTests', others still inactive; the 2 ExtensionTests failed with ShimIpPbxRemoteDestination-TypeLoadException  
 23    |  38/65 passed    |  same source code as #22, notice NOTHING has changed, however 1 unit test was not even run in 'ExtensionTests'; however the second succeeds; all unit tests in 'UserTests' using MS Fakes fail with a StubClient-TypeLoadException   
 24    |  38/65 passed    |  same source code as #22  

new day - no more luck :-(

  1     |  37/64 passed    | uncommented the 2 'ExtensionTests', so same code as #19
  2     |  37/64 passed    | so same code as #1
  3     |  64/64 passed    | so same code as #1; suddenly they all work again  

Notice that beside the mentioned changes nobody has changed the source code.
All build were made using a detailed log-level ; Shim Diagnostic="true";
only explicitly needed shims/stubs are included in .fakes-file (otherwhise MSBUILD was failing using exit-code 1).

It all seems quite magic to me, possibly somebody has already experienced the same problems or somebody has a hint.

Thanks in advance

Answer 1

I think I know where the issue lies, the TypeNotFoundException was correct, at least the corresponding types were not generated (as shown by Reflector), this made me thinking. More exactly there is only one generated fake-file generated per assembly lying under [buildName]\Binaries- and as both of my Assemblies faked the same Assembly (backend.dll) - this resulted most probably in a race condition - the one assembly which was built later than the other had its final shims/stubs generated; this doesn't clarify all problems though.

Anyway the reason why this doesn't show up on a regular VS build is because the assemblies are generated in their respective [assembly]\FakeAssemblies directory, not influencing any other unit test (the way it is supposed to be).

So my idea to circumvent this problem was to put ALL shims/stubs used by both test-assemblies into both .fakes-configuration-files; reflector showed that really all needed shims/stubs got generated this time, however the build failed with

Exception Message: MSBuild error 1 has ended this build. 

With no more information in any (detailed) log. This problem is caused by not using a generated shim/stub-object specified in the .fakes-file (and of course this is also a problem just shown by the build server, would be too boring otherwise ;-)).

So the final solution I came up with is to put ALL code that makes use of shims/stubs into the same assembly, where only 1 .fakes-file configures the fake-Assembly.

Now it is working fine :-), but I think the MS-Dev-Team has still something to do on here, haven't tested it with TFS2013 though; gonna submit a bug anyway (as I've discovered 2 already)

Hope this helps somebody with the same problem.