Run code once before and after ALL tests in xUnit.net

Question

TL;DR - I'm looking for xUnit's equivalent of MSTest's AssemblyInitialize (aka the ONE feature it has that I like).

Specifically I'm looking for it because I have some Selenium smoke tests which I would like to be able to run with no other dependencies. I have a Fixture that will launch IisExpress for me and kill it on disposal. But doing this before every test hugely bloats runtime.

I would like to trigger this code once at the start of testing, and dispose of it (shutting down the process) at the end. How could I go about doing that?

Even if I can get programmatic access to something like "how many tests are currently being run" I can figure something out.

Answer 1

As of Nov 2015 xUnit 2 is out, so there is a canonical way to share features between tests. It is documented here.

Basically you'll need to create a class doing the fixture:

    public class DatabaseFixture : IDisposable
    {
        public DatabaseFixture()
        {
            Db = new SqlConnection("MyConnectionString");

            // ... initialize data in the test database ...
        }

        public void Dispose()
        {
            // ... clean up test data from the database ...
        }

        public SqlConnection Db { get; private set; }
    }

A dummy class bearing the CollectionDefinition attribute. This class allows Xunit to create a test collection, and will use the given fixture for all test classes of the collection.

    [CollectionDefinition("Database collection")]
    public class DatabaseCollection : ICollectionFixture<DatabaseFixture>
    {
        // This class has no code, and is never created. Its purpose is simply
        // to be the place to apply [CollectionDefinition] and all the
        // ICollectionFixture<> interfaces.
    }

Then you need to add the collection name over all your test classes. The test classes can receive the fixture through the constructor.

    [Collection("Database collection")]
    public class DatabaseTestClass1
    {
        DatabaseFixture fixture;

        public DatabaseTestClass1(DatabaseFixture fixture)
        {
            this.fixture = fixture;
        }
    }

It's a bit more verbose than MsTests AssemblyInitialize since you have to declare on each test class which test collection it belongs, but it's also more modulable (and with MsTests you still need to put a TestClass on your classes)

Note: the samples have been taken from the documentation.

Answer 2

To execute code on assembly initialize, then one can do this (Tested with xUnit 2.3.1)

using Xunit.Abstractions;
using Xunit.Sdk;

[assembly: Xunit.TestFramework("MyNamespace.MyClassName", "MyAssemblyName")]

namespace MyNamespace
{   
   public class MyClassName : XunitTestFramework
   {
      public MyClassName(IMessageSink messageSink)
        :base(messageSink)
      {
        // Place initialization code here
      }

      public new void Dispose()
      {
        // Place tear down code here
        base.Dispose();
      }
   }
}

See also https://github.com/xunit/samples.xunit/tree/master/AssemblyFixtureExample

Answer 3

Create a static field and implement a finalizer.

You can use the fact that xUnit creates an AppDomain to run your test assembly and unloads it when it's finished. Unloading the app domain will cause the finalizer to run.

I am using this method to start and stop IISExpress.

public sealed class ExampleFixture
{
    public static ExampleFixture Current = new ExampleFixture();

    private ExampleFixture()
    {
        // Run at start
    }

    ~ExampleFixture()
    {
        Dispose();
    }

    public void Dispose()
    {
        GC.SuppressFinalize(this);

        // Run at end
    }        
}

Edit: Access the fixture using ExampleFixture.Current in your tests.

Answer 4

It's not possible to do in the framework today. This is a feature planned for 2.0.

In order to make this work before 2.0, it would require you to perform significant re-architecture on the framework, or write your own runners that recognized your own special attributes.

Answer 5

I use AssemblyFixture (NuGet).

What it does is it provides an IAssemblyFixture<T> interface that is replacing any IClassFixture<T> where you want the object's lifetime to be as the testing assembly.

Example:

public class Singleton { }

public class TestClass1 : IAssemblyFixture<Singleton>
{
  readonly Singletone _Singletone;
  public TestClass1(Singleton singleton)
  {
    _Singleton = singleton;
  }

  [Fact]
  public void Test1()
  {
     //use singleton  
  }
}

public class TestClass2 : IAssemblyFixture<Singleton>
{
  readonly Singletone _Singletone;
  public TestClass2(Singleton singleton)
  {
    //same singleton instance of TestClass1
    _Singleton = singleton;
  }

  [Fact]
  public void Test2()
  {
     //use singleton  
  }
}

Answer 6

I was quite annoyed for not having the option to execute things at the end of all the xUnit tests. Some of the options here are not as great, as they involve changing all your tests or putting them under one collection (meaning they get executed synchronously). But Rolf Kristensen's answer gave me the needed information to get to this code. It's a bit long, but you only need to add it into your test project, no other code changes necessary:

using Siderite.Tests;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Text;
using Xunit;
using Xunit.Abstractions;
using Xunit.Sdk;

[assembly: TestFramework(
    SideriteTestFramework.TypeName,
    SideriteTestFramework.AssemblyName)]

namespace Siderite.Tests
{
    public class SideriteTestFramework : ITestFramework
    {
        public const string TypeName = "Siderite.Tests.SideriteTestFramework";
        public const string AssemblyName = "Siderite.Tests";
        private readonly XunitTestFramework _innerFramework;

        public SideriteTestFramework(IMessageSink messageSink)
        {
            _innerFramework = new XunitTestFramework(messageSink);
        }

        public ISourceInformationProvider SourceInformationProvider
        {
            set
            {
                _innerFramework.SourceInformationProvider = value;
            }
        }

        public void Dispose()
        {
            _innerFramework.Dispose();
        }

        public ITestFrameworkDiscoverer GetDiscoverer(IAssemblyInfo assembly)
        {
            return _innerFramework.GetDiscoverer(assembly);
        }

        public ITestFrameworkExecutor GetExecutor(AssemblyName assemblyName)
        {
            var executor = _innerFramework.GetExecutor(assemblyName);
            return new SideriteTestExecutor(executor);
        }

        private class SideriteTestExecutor : ITestFrameworkExecutor
        {
            private readonly ITestFrameworkExecutor _executor;
            private IEnumerable<ITestCase> _testCases;

            public SideriteTestExecutor(ITestFrameworkExecutor executor)
            {
                this._executor = executor;
            }

            public ITestCase Deserialize(string value)
            {
                return _executor.Deserialize(value);
            }

            public void Dispose()
            {
                _executor.Dispose();
            }

            public void RunAll(IMessageSink executionMessageSink, ITestFrameworkDiscoveryOptions discoveryOptions, ITestFrameworkExecutionOptions executionOptions)
            {
                _executor.RunAll(executionMessageSink, discoveryOptions, executionOptions);
            }

            public void RunTests(IEnumerable<ITestCase> testCases, IMessageSink executionMessageSink, ITestFrameworkExecutionOptions executionOptions)
            {
                _testCases = testCases;
                _executor.RunTests(testCases, new SpySink(executionMessageSink, this), executionOptions);
            }

            internal void Finished(TestAssemblyFinished executionFinished)
            {
                // do something with the run test cases in _testcases and the number of failed and skipped tests in executionFinished
            }
        }


        private class SpySink : IMessageSink
        {
            private readonly IMessageSink _executionMessageSink;
            private readonly SideriteTestExecutor _testExecutor;

            public SpySink(IMessageSink executionMessageSink, SideriteTestExecutor testExecutor)
            {
                this._executionMessageSink = executionMessageSink;
                _testExecutor = testExecutor;
            }

            public bool OnMessage(IMessageSinkMessage message)
            {
                var result = _executionMessageSink.OnMessage(message);
                if (message is TestAssemblyFinished executionFinished)
                {
                    _testExecutor.Finished(executionFinished);
                }
                return result;
            }
        }
    }
}

The highlights:

• assembly: TestFramework instructs xUnit to use your framework, which proxies to the default one
• SideriteTestFramework also wraps the executor into a custom class that then wraps the message sink
• in the end, the Finished method is executed, with the list of tests run and the result from the xUnit message

More work could be done here. If you want to execute stuff without caring about the tests run, you could inherit from XunitTestFramework and just wrap the message sink.

Answer 7

You can use IUseFixture interface to make this happen. Also all of your test must inherit TestBase class. You can also use OneTimeFixture directly from your test.

public class TestBase : IUseFixture<OneTimeFixture<ApplicationFixture>>
{
    protected ApplicationFixture Application;

    public void SetFixture(OneTimeFixture<ApplicationFixture> data)
    {
        this.Application = data.Fixture;
    }
}

public class ApplicationFixture : IDisposable
{
    public ApplicationFixture()
    {
        // This code run only one time
    }

    public void Dispose()
    {
        // Here is run only one time too
    }
}

public class OneTimeFixture<TFixture> where TFixture : new()
{
    // This value does not share between each generic type
    private static readonly TFixture sharedFixture;

    static OneTimeFixture()
    {
        // Constructor will call one time for each generic type
        sharedFixture = new TFixture();
        var disposable = sharedFixture as IDisposable;
        if (disposable != null)
        {
            AppDomain.CurrentDomain.DomainUnload += (sender, args) => disposable.Dispose();
        }
    }

    public OneTimeFixture()
    {
        this.Fixture = sharedFixture;
    }

    public TFixture Fixture { get; private set; }
}

EDIT: Fix the problem that new fixture create for each test class.

Answer 8

Does your build tool provide such a feature?

In the Java world, when using Maven as a build tool, we use the appropriate phases of the build lifecycle. E.g. in your case (acceptance tests with Selenium-like tools), one can make good use of the pre-integration-test and post-integration-test phases to start/stop a webapp before/after one's integration-tests.

I'm pretty sure the same mechanism can be set up in your environment.

Answer 9

The method described by Jared Kells does not work under Net Core, because, well it is not guaranteed that finalizers will be called. And, in fact, it is not called for the code above. Please, see:

Why does the Finalize/Destructor example not work in .NET Core?

https://github.com/dotnet/runtime/issues/16028

https://github.com/dotnet/runtime/issues/17836

https://github.com/dotnet/runtime/issues/24623

So, based on the great answer above, here is what I ended up doing (replace saving to file as necessary):

public class DatabaseCommandInterceptor : IDbCommandInterceptor
{
    private static ConcurrentDictionary<DbCommand, DateTime> StartTime { get; } = new();

    public void ReaderExecuted(DbCommand command, DbCommandInterceptionContext<DbDataReader> interceptionContext) => Log(command, interceptionContext);

    public void NonQueryExecuted(DbCommand command, DbCommandInterceptionContext<int> interceptionContext) => Log(command, interceptionContext);

    public void ScalarExecuted(DbCommand command, DbCommandInterceptionContext<object> interceptionContext) => Log(command, interceptionContext);

    private static void Log<T>(DbCommand command, DbCommandInterceptionContext<T> interceptionContext)
    {
        var parameters = new StringBuilder();

        foreach (DbParameter param in command.Parameters)
        {
            if (parameters.Length > 0) parameters.Append(", ");
            parameters.Append($"{param.ParameterName}:{param.DbType} = {param.Value}");
        }

        var data = new DatabaseCommandInterceptorData
        {
            CommandText = command.CommandText,
            CommandType = $"{command.CommandType}",
            Parameters = $"{parameters}",
            Duration = StartTime.TryRemove(command, out var startTime) ? DateTime.Now - startTime : TimeSpan.Zero,
            Exception = interceptionContext.Exception,
        };

        DbInterceptorFixture.Current.LogDatabaseCall(data);
    }

    public void NonQueryExecuting(DbCommand command, DbCommandInterceptionContext<int> interceptionContext) => OnStart(command);
    public void ReaderExecuting(DbCommand command, DbCommandInterceptionContext<DbDataReader> interceptionContext) => OnStart(command);
    public void ScalarExecuting(DbCommand command, DbCommandInterceptionContext<object> interceptionContext) => OnStart(command);

    private static void OnStart(DbCommand command) => StartTime.TryAdd(command, DateTime.Now);
}

public class DatabaseCommandInterceptorData
{
    public string CommandText { get; set; }
    public string CommandType { get; set; }
    public string Parameters { get; set; }
    public TimeSpan Duration { get; set; }
    public Exception Exception { get; set; }
}

/// <summary>
/// All times are in milliseconds.
/// </summary>
public record DatabaseCommandStatisticalData
{
    public string CommandText { get; }
    public int CallCount { get; init; }
    public int ExceptionCount { get; init; }
    public double Min { get; init; }
    public double Max { get; init; }
    public double Mean { get; init; }
    public double StdDev { get; init; }

    public DatabaseCommandStatisticalData(string commandText)
    {
        CommandText = commandText;
        CallCount = 0;
        ExceptionCount = 0;
        Min = 0;
        Max = 0;
        Mean = 0;
        StdDev = 0;
    }

    /// <summary>
    /// Calculates k-th moment for n + 1 values: M_k(n + 1)
    /// based on the values of k, n, mkn = M_k(N), and x(n + 1).
    /// The sample adjustment (replacement of n -> (n - 1)) is NOT performed here
    /// because it is not needed for this function.
    /// Note that k-th moment for a vector x will be calculated in Wolfram as follows:
    ///     Sum[x[[i]]^k, {i, 1, n}] / n
    /// </summary>
    private static double MknPlus1(int k, int n, double mkn, double xnp1) =>
        (n / (n + 1.0)) * (mkn + (1.0 / n) * Math.Pow(xnp1, k));

    public DatabaseCommandStatisticalData Updated(DatabaseCommandInterceptorData data) =>
        CallCount == 0
            ? this with
            {
                CallCount = 1,
                ExceptionCount = data.Exception == null ? 0 : 1,
                Min = data.Duration.TotalMilliseconds,
                Max = data.Duration.TotalMilliseconds,
                Mean = data.Duration.TotalMilliseconds,
                StdDev = 0.0,
            }
            : this with
            {
                CallCount = CallCount + 1,
                ExceptionCount = ExceptionCount + (data.Exception == null ? 0 : 1),
                Min = Math.Min(Min, data.Duration.TotalMilliseconds),
                Max = Math.Max(Max, data.Duration.TotalMilliseconds),
                Mean = MknPlus1(1, CallCount, Mean, data.Duration.TotalMilliseconds),
                StdDev = Math.Sqrt(
                    MknPlus1(2, CallCount, Math.Pow(StdDev, 2) + Math.Pow(Mean, 2), data.Duration.TotalMilliseconds)
                    - Math.Pow(MknPlus1(1, CallCount, Mean, data.Duration.TotalMilliseconds), 2)),
            };

    public static string Header { get; } =
        string.Join(TextDelimiter.VerticalBarDelimiter.Key,
            new[]
            {
                nameof(CommandText),
                nameof(CallCount),
                nameof(ExceptionCount),
                nameof(Min),
                nameof(Max),
                nameof(Mean),
                nameof(StdDev),
            });

    public override string ToString() =>
        string.Join(TextDelimiter.VerticalBarDelimiter.Key,
            new[]
            {
                $"\"{CommandText.Replace("\"", "\"\"")}\"",
                $"{CallCount}",
                $"{ExceptionCount}",
                $"{Min}",
                $"{Max}",
                $"{Mean}",
                $"{StdDev}",
            });
}

public class DbInterceptorFixture
{
    public static readonly DbInterceptorFixture Current = new();
    private bool _disposedValue;
    private ConcurrentDictionary<string, DatabaseCommandStatisticalData> DatabaseCommandData { get; } = new();
    private static IMasterLogger Logger { get; } = new MasterLogger(typeof(DbInterceptorFixture));

    /// <summary>
    /// Will run once at start up.
    /// </summary>
    private DbInterceptorFixture()
    {
        AssemblyLoadContext.Default.Unloading += Unloading;
    }

    /// <summary>
    /// A dummy method to call in order to ensure that static constructor is called
    /// at some more or less controlled time.
    /// </summary>
    public void Ping()
    {
    }

    public void LogDatabaseCall(DatabaseCommandInterceptorData data) =>
        DatabaseCommandData.AddOrUpdate(
            data.CommandText,
            _ => new DatabaseCommandStatisticalData(data.CommandText).Updated(data),
            (_, d) => d.Updated(data));

    private void Unloading(AssemblyLoadContext context)
    {
        if (_disposedValue) return;
        GC.SuppressFinalize(this);
        _disposedValue = true;
        SaveData();
    }

    private void SaveData()
    {
        try
        {
            File.WriteAllLines(
                @"C:\Temp\Test.txt",
                DatabaseCommandData
                    .Select(e => $"{e.Value}")
                    .Prepend(DatabaseCommandStatisticalData.Header));
        }
        catch (Exception e)
        {
            Logger.LogError(e);
        }
    }
}

and then register DatabaseCommandInterceptor once somewhere in the tests:

DbInterception.Add(new DatabaseCommandInterceptor());

I also prefer calling DbInterceptorFixture.Current.Ping() in the base test class, though I don't think that this is needed.

The interface IMasterLogger is just a strongly typed wrapper around log4net, so just replace it with your favorite one.

The value of TextDelimiter.VerticalBarDelimiter.Key is just '|' and it sits in what we call a closed set.

PS If I screwed up with statistics, please, comment and I will update the answer.