Different Code Analysis results on the same rule set

Question

When running the exact same rule set on our local machines and on the build server on the same code base (TFS on Visual Studio Online), we get different results.

Code Analysis output from local:

Running Code Analysis...
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Team Tools\Static Analysis Tools\FxCop\FxCopCmd.exe  
/outputCulture:1033 
/ruleSet:"=C:\Workspaces\ProjectName\CodeAnalysis\CodeAnalysisRuleSet.ruleset" 
/rulesetdirectory:"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Team Tools\Static Analysis Tools\\Rule Sets" 
/rule:"-C:\Program Files (x86)\Microsoft Visual Studio 12.0\Team Tools\Static Analysis Tools\FxCop\\Rules" 
/searchgac 
/ignoreinvalidtargets 
/forceoutput 
/successfile 
/ignoregeneratedcode 
/saveMessagesToReport:Active 
/timeout:120 

Code Analysis output from build server:

Running Code Analysis...
C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Static Analysis Tools\FxCop\FxCopCmd.exe  
/outputCulture:1033 
/ruleSet:"=C:\a\Sources\ProjectName\Trunk\SharedResources\..\..\CodeAnalysis\CodeAnalysisRuleSet.ruleset" 
/rulesetdirectory:"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Static Analysis Tools\\Rule Sets" 
/rule:"-C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Static Analysis Tools\FxCop\\Rules" 
/searchgac 
/ignoreinvalidtargets 
/forceoutput 
/successfile 
/ignoregeneratedcode 
/saveMessagesToReport:Active 
/timeout:120 
MSBUILD : error : CA1034 : Microsoft.Design : Do not nest type 'SpatialFunctions.RelativeStopType'. Alternatively, change its accessibility so that it is not externally visible.
  Code Analysis Complete -- 1 error(s), 0 warning(s)

The build server picks up an issue which was missed on the local analysis (CA1034).

The only difference that I can see is that TFS is using MS Visual Studio 11 tools (whereas we are developing with VS 12.0). I can't imagine why this should make a difference.

We are using Visual Studio 2013.

Answer 1

Your build server is using the FxCop binaries from Visual Studio 2012, not those of 2013. Each time a new version of Visual Studio is released there have been some updates to the rules to fix bugs and there have been a few rules that got added. There's also a difference in certain code generation scenario's between .NET SDK versions.

Given this knowledge, I am not surprised that there are different findings from the different versions of the tools. How to fix it you might ask? Install Visual Studio 2013 on the build server and make sure your "Run MSBuild for Projects" activity in team build points to the correct toolsdirectory (the MsBuild one that ships with Visual Studio 2013).

Or: Make sure you're using TFS 2013 and Team Build 2013, which should figure this out by itself if both versions of Visual Studio are installed.

Answer 2

The CA1034 rule has been changed in VS2013 to ignore nested enums. If you don't want nested enums ignored, you would need to add a custom rule to detect them under VS2013. If you don't mind ignoring them, then building under VS2013 on the build server would seem to be the easiest way to resolve the discrepancy. (Regardless of how you decide to handle the discrepancy, building under the same VS version on developer machines and the build server would be very much recommended.)