Visual Studio 2012 test categories hierarchy (Test Explorer)

Question

I am testing a quite big project (C#, VS2012), and I need to arrange my unit test in test hierarchy (eg.: now I have 43 test cases). I do really need the hierarchy.

I have test categories defined already, and the test explorer shows test cases by traits. I have categories in this way (one test have several categories)

• TestCase01: MainTestType, SubTestType, SubsubTestType
• ...
• TestCase10: MainTestType, SubTestType, SubsubTestType
• TestCase11: MainTestType, SubTestType2, SubsubTestType2
• ...
• TestCase15: MainTestType, SubTestType2, SubsubTestType2

Defined like this:

    [TestMethod]
    [TestCategory("MainTestType")]
    [TestCategory("SubTestType")]
    [TestCategory("SubsubTestType")]
    public void MyTestCase()
    { /* etc. */

But Test Explorer shows the next:

• MainTestType: all tests having category MainTestType
• SubTestType: all tests having category SubTestType
• etc...

So I really miss the hierarchy. I have tried "Cat1\Cat2\Cat3" or even with /. But no hierarchy displayed. Do you know how to do it, or a free addon which can do it for me?

I also will need these type of categorization, because we run often tests from command line, and mstest.exe can run tests for one category (eg all MainTestType, or SubTestType). (I stick to mstest because half of the team uses vs2010). But the solution is enough for vs2012.

Thank you in advance.

Answer 1

As what I searched for is not supported at the moment, I made the next workaround:

• build the project
• run MSText for all tests -> .trx output
• simple winforms/wpf program, which parses .trx, gets the test cases and displays them in a treeview
• now we can run mstest from this application for the selected node -> creates .trx output (which can be opened in VS)

I used .trx because that way I don't have to do parse the assembly, mstest.exe does it. Test categories are made this way:

[TestCategory("MainTestType")]
[TestCategory("MainTestType/SubTestType")]
[TestCategory("MainTestType/SubTestType/SubsubTestType")]

So the is the workaround which is simple, only one binary and the developers can use too. The problem with playlist was that they aren't hierarchical neither.

Answer 2

Test Explorer only shows groups following the TestFilter/InnerTests layout.

The New Test Explorer Tab is a generic container. Using specific adapters/addon/plugin you can integrate other test frameworks (NUnit, xUnit, Qunit, ...) into Test Explorer ... but these adapters have to follow the test explorer interface.

So, it's not natively supported by the Test Explorer Tab, I didn't find in the past a way to extend the test explorer tab.

Maybe another test framework will allow you to have a hierarchy but this will be in another custom Tab and with another attribute (TestCategory is a MSTest attribute).

Just for information, you can alos combine traits in the Test Explorer Window in this way

Trait:"MainTestType1"  Trait:"SubTestType3"

Answer 3

For grouping in VS2010 use .vsmdi files that could be opened in Test List Editor.

As far as VS2012:

Visual Studio 2012 Update 1 http://www.microsoft.com/visualstudio/eng/visual-studio-update has added enhancements to support both grouping and filtering by Project and Traits (category). Use of these features is detailed in this blog post, http://blogs.msdn.com/b/visualstudioalm/archive/2012/11/09/how-to-manage-unit-tests-in-visual-studio-2012-update-1-part-1-using-traits-in-the-unit-test-explorer.aspx

Group and filter by Class is completed and will be available in Update 2.

I suggest you to move your categories to namespace, so you would be able to see flat hierarchy in TestView window in FullClassName column.

Answer 4

You could cheat a bit by making your level 2 trait a concatenation of your first two levels' values. So if level 1 is Animal and level 2 is Mammal, you could have a Trait "Animal.Mammal". In the Test Explorer, you'd get a node for Level1 [Animal], another for Level1 [Plant], and then one for Level2 [Animal.Tiger]. I believe the filter supports substring matches, so that's one way to sort it. You still don't get a tree view (although if you were bound and determined you could try extended ascii art pipe characters like the old DOS tree command).

You also still have the problem that you can't merge different traits (such that all the birds would be together in the listing all the way down).

On the other hand, enforcing a hierarchy structure while allowing n-tuples of traits would be awkward. It would have to restrict items from being in more than one path from the top of the hierarchy, detect and report cycles, use a tree control on top of (or matrixed with) what's there now. Also, when you refactor code and want to keep the tests arranged in parallel, too much existing structure could cause headaches.