Make TeamCity build fail if tests are not run

Question

We have a setup where TeamCity builds the software and the runs the unit tests on it. We use a powershell script inspired by this answer, which runs the unit tests with vstest and reports code coverage with dotcover.

Sometimes, developers make changes that cause the script to not run tests at all. Instead of failing the build, TeamCity reports the build as successful, but doesn't mention tests (since they weren't run).

I would like to have the build fail if the tests are not run.

How do I accomplish this?

Answer 1

You can add build failure condition on metric change (see Build Failure Conditions step of build configuration administration area). There is metric called "number of tests". Set it to 0, to fail build if no tests ran.

Answer 2

I took a look at the powershell script you referenced. I see two obvious reasons why tests might not run:

# Get list of folders with Test DLLs in them matching pattern *Tests\bin
$testFolders = Get-ChildItem -Recurse -Force $rootDirectory | 
  Where-Object { ($_.PSIsContainer -eq $true) -and (($_.FullName -like "*Tests\bin\" + $vsConfigName) -or ($_.FullName -like "*Tests\bin\x64\" + $vsConfigName)) }

If no folders match the pattern in this gci then no tests will be run.

#grab the testing DLLs from the folder which match pattern *Tests.dll
$testDlls = Get-ChildItem -Force $folder.FullName -File |
  Where-Object { $_.Name -like "*Tests.dll" }

Similarly if no dlls match the pattern no test will be run.

Besides that the script should really have a try catch around it, then you can throw if you either don't find any test folders or any test dlls. I've edited the code sample you referenced to show this:

#rootDirectory is the base directory to search for Test Projects. (Most likely your solution directory)
#configName is a string name that the code coverage output files will be placed in
#filters is a list of dotCover filters to be added to the /Filters argument
#vsConfigName is the configuration folder to find the Test DLLs in
Param([string]$rootDirectory,[string]$configName,[string]$filters,[string]$vsConfigName)

$vstestconsolepath = "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"
$dotcoverpath = "C:\BuildAgent\tools\dotCover\dotCover.exe"
$dotcovertargetexecutable = "/TargetExecutable=" + $vstestconsolepath
$dotcoveroutput = "/Output=" + $configName + "/coverage.dcvr"
$dotcoverfilters = "/Filters=" + $filters

try
{
  # Get list of folders with Test DLLs in them matching pattern *Tests\bin
  $testFolders = Get-ChildItem -Recurse -Force $rootDirectory | Where-Object { ($_.PSIsContainer -eq $true) -and (($_.FullName -like "*Tests\bin\" + $vsConfigName) -or ($_.FullName -like "*Tests\bin\x64\" + $vsConfigName)) } | Select-Object
  if ($testFolder -eq $null) 
  {
    throw [Exception] "No test folders found."
  }

  foreach ($folder in $testFolders)
  {
      #look for Fakes DLLs. If we find one we can't do code coverage on this test assembly
      $fakesDLLs = Get-ChildItem -Recurse -Force $folder.FullName -File | Where-Object { $_.Name -like "*Fakes.dll" } | Select-Object

      #grab the testing DLLs from the folder which match pattern *Tests.dll
      $testDlls = Get-ChildItem -Force $folder.FullName -File | Where-Object { $_.Name -like "*Tests.dll" } | Select-Object

      foreach ($dll in $testDlls)
      {
          if ($fakesDLLs.length -eq 0)
          {
              $arr += @($dll.FullName)
          }
          else
          {
              $fakesArr += @($dll.FullName)
          }
      }
  }

  if ($arr -eq $null) 
  {
    throw [Exception] "No test dlls found."
  }

...

}
catch [Exception]
{
  Write-Host($_.Exception.GetType().FullName)
  Write-Host($_.Exception.Message)
  Write-Host($_.Exception.StackTrace)
  exit 1
}

Of course you don't necessarily need to go that far, you could just exit 1 rather than throwing with the try catch. And of course you shouldn't really throw Exception but that's off topic.