How to run tests in a playlist file from mstest command line

Question

Is there a way I can use the playlist file to run tests from commandline using MSTest.exe? I tried using following command line but it fails with and error -

mstest.exe /testmetadata:test.playlist

The file 'test.playlist' has unknown format and cannot be converted to the current version.

Note that mstest version is 11.0.50727.1 and I am using VS 2012

my playlist file just contains couple of XML elements

<Playlist Version="1.0">
  <Add Test="MyTest" />
  <Add Test="AnotherTest" />
</Playlist>

Answer 1

mstest.exe has been deprecated. For Visual Studio 2012 SP1 and above we use vstest.console.exe. It still won't run your playlist file. There is a feature request open with Microsoft.

We're in a bad state right now with test lists also deprecated. The only way I can see to discriminate between different tests is to use Test Categories

Another option would be to switch to a different unit testing framework.

I wish there were a better answer.

Answer 2

As Nate Zaugg mentioned, there isn't currently a way to do it, but it's fairly easy to generate a list of tests programmatically.

The playlist file is just XML. It has a root node with child nodes. You can read the "Test" attribute of each node to get the FullyQualifiedNames of each of the tests you want to run. You can probably strip out the namespace and class before the test names at this point.

If you're able to use vstest.console.exe (which you should, with VS 2012), you can follow the instructions below. vstest.console.exe has the optional parameter "TestCaseFilter" which you can use to call out each test mentioned in the playlist. It's not particularly elegant, but it should work.

So, for some generic tests "MethodName1", "MethodName2", and "MethodName3", that are in the myTestFile.dll and myOtherTestFile.dll you would generate the following command:

vstest.console.exe myTestFile.dll myOtherTestFile.dll /Tests:MethodName1,MethodName2,MethodName3"

Answer 3

You could use .orderedtest instead of .playlist

Ordered tests can be created and edited in VS2013. The format is otherwise similiar to .playlist but it contains links to test GUIDs so its more complicated to modify programmatically.

From command line, run with:

MSTest.exe /testcontainer:mylist.orderedtest

Not sure if it works in VS2012.

Answer 4

It looks like with VSTest we now have a TestCaseFilter option with TestCategory and Priority. I'm still looking into it, but hopefully this helps someone else.