How do I skip specific tests in xUnit based on current platform

Question

• I have an assembly that I've built on Windows
• I want to run the xUnit tests on mono in Linux.

However, I have found that while 400 of these tests can run (in order), that certain tests either hang the xUnit runner, or bring it down entirely.

I don't care if certain tests are not able to run on Linux, certain tests are to do with the DTC and some unmanaged gumph that we don't need to support there.

What I do want however, is to apply an ignore to those tests, and have the fact that the test was ignored flagged properly in the build output.

The question can be boiled down to I guess a number of possible solutions

• How do I run specific tests in xUnit via the console runner? (I haven't found documentation to this end, maybe I'm just not looking hard enough)
• Is it possible to go the other way and say "Here is an assembly, please ignore these specific tests though"
• Having an attribute on those tests has been suggested a better way, to formally document that these tests are platform specific - is this possible?

If I could avoid modifying the original code too much that would be grand, as the code isn't really mine to change, and applying lots of cross-platform hacks probably won't go down too well.

Answer 1

I would avoid externalising skipping tests (i.e. a config/command file if it's possible). This somewhat goes against making the tests easy to run and trustworthy. Making the tests ignored in code is the safest approach when other people start to get involved.

I could see a number of options, here are two that involve modification of existing code.

Option 1 - Most intrusive, compile time platform detection

In the VS Solution, define another configuration that defines a precompiler flag MONOWIN (just so that it's explicitly a flag the says that it is for code compiled on Windows for use on Mono).

Then define an attribute that will make the test ignored when compiled for Mono:

public class IgnoreOnMonoFactAttribute : FactAttribute {
#if MONOWIN
    public IgnoreOnMonoFactAttribute() {
        Skip = "Ignored on Mono";
    }
#endif
}

It's actually hard to find any advantages to this method as it involves mocking with the original solution and adds another confiration that needs to be supported.

Option 2 - somewhat intrusive - runtime platform detection

Here is a similar solution to option1, except no separate configuration is required:

public class IgnoreOnMonoFactAttribute : FactAttribute {

    public IgnoreOnMonoFactAttribute() {
        if(IsRunningOnMono()) {
            Skip = "Ignored on Mono";
        }
    }
    /// <summary>
    /// Determine if runtime is Mono.
    /// Taken from http://stackoverflow.com/questions/721161
    /// </summary>
    /// <returns>True if being executed in Mono, false otherwise.</returns>
    public static bool IsRunningOnMono() {
        return Type.GetType("Mono.Runtime") != null;
    }
}

Note 1

xUnit runner will run a method twice if it is marked with [Fact] and [IgnoreOnMonoFact]. (CodeRush doesn't do that, in this case I assume xUnit is correct). This means that any tests methods must have [Fact] replaced with [IgnoreOnMonoFact]

Note 2

CodeRush test runner still ran the [IgnoreOnMonoFact] test, but it did ignore the [Fact(Skip="reason")] test. I assume it is due to CodeRush reflecting xUnit and not actually running it with the aid of xUnit libraries. This works fine with xUnit runner.

Answer 2

XUnit v2.0 is now available. Skippable tests are supported by it directly. Use:

[Fact (Skip = "specific reason")]

Answer 3

There is a new options now.

Add Nuget Package SkippableFact, which allows you to use [SkippableFact] instead of [Fact] and you can use Skip.<xyz> within a Tests to dynamically Skip the Test during runtime.

Example:

[SkippableFact]
public void SomeTestForWindowsOnly()
{
    Skip.IfNot(Environment.IsWindows);

    // Test Windows only functionality.
}

Answer 4

This would be an ideal use of traits, but unfortunately, neither the command line nor the xml project file supports filtering based on traits. Would be worth adding an issue to the codeplex site for this.

Answer 5

This is now solved in 1.8 - you can filter on Traits. See this issue log.

Update: Traits work with the console runner but not MSBuild, I've added a feature request for this support.