Are there "Dynamic Playlists" of unit tests in Visual Studio?

Question

When working with unit tests in Visual Studio, it's possible to compile playlists that group the tests. This is useful, for instance, when developing a new feature for which we want to run only the related tests and not all the existing ones.

However, it seems to me the playlists can only be maintained "manually", by explicitly adding/removing unit tests.

It's better than nothing, but it would be great if I could some how "tag" my unit tests depending on some criterion (e.g. feature, test duration, etc.) and generate the playlists dynamically.

Is there any way to do something similar in Visual Studio with the default testing framework?

Answer 1

You can run certain tests contextually in relation to the current cursor position by right clicking and then "Run Tests", or hitting CTRL+R then T.

• If the cursor is within the body of a test, only that test will be run.
• If the cursor is within the body of a test class, only the tests in that class will be run.
• If the cursor is within a namespace, all tests in that namespace will be run.

See MSDN documentation on the subject.

Alternatively you can filter the test explorer in various ways. I often use the Project flag to exclude certain projects for instance.

Answer 2

The ability to create a playlist which is defined dynamically was added with update 16.7.0 to Visual Studio 2019.

According to those notes, the playlist User Interface now contains checkboxes for the various selection criteria. The underlying XML seems to be a cascade of <Rule> elements.

To access the new user interface, it may be necessary to click the new pencil icon (tooltip: edit) in the Test Explorer toolbar, and confirm that the playlist may be updated to the new format.

From the source:

You can also use traits to define a dynamic group by editing the playlist xml file directly.

As of 2020-12-07, I finally got this to work. For the goal to run any test that includes the MSTest Attribute [TestCategory("SchemaUpdateBasic")], possibly with other instances of [TestCategory("...")]

<Playlist Version="2.0">
  <Rule Name="Includes" Match="Any">
    <Property Name="Trait" Value="SchemaUpdateBasic" />
  </Rule>
</Playlist>

As of Visual Studio version 16.8.2, this worked as desired. Prior versions resulted, for me, in the correct set of tests appearing in the Test Explorer window, but clicking "Run All" yielding this in Test Output

---------- Starting test discovery for requested test run ---------- Test run will use DLL(s) built for framework
...
========== Test discovery finished: 0 Tests found in 2.6 sec ========== No tests found to run.

Source: https://learn.microsoft.com/en-us/visualstudio/releases/2019/release-notes#--visual-studio-2019-version-1670

Also https://github.com/MicrosoftDocs/visualstudio-docs/issues/6012 exists as a request for documentation enhancement.

Answer 3

In VS2017, you can group tests by Namespace, Class, Duration, Outcome, Traits or Project. Probably was there before, but I can't check first hand right now.

To do so, in the Test Explorer window, click on the second icon then whichever choice you want. By default, they're grouped by Outcome.

If you want to use Traits, add the Trait name/value attribute to your individual test cases:

[Trait("Foo", "Value1")]
[Trait("Foo", "Value2")]
[Trait("Bar", "Value")]
public void MyClass_MyMethod_WhenThing_Outcome()
{
    // ...
}