MBUnit test matrix optimization-performace problems in automated ui tests

Question

We're currently using MBUnit for both unit testing and UI testing. For UI testing setup cost for test matrix axes are pretty high (login, browser instance, navigate to page etc). In order to avoid setting up these for each test case we are partly relying on AssemblyFixture to manage some of them.

However because it's not possible to filter out certain cases where they are not applicable to certain combination, it's not possible for us to really use such optimization. So currently we are doing some of the setup per test-case, horribly inefficient.

We could put if statements inside test code to check for correct combinations but we don't desire that either. It pollutes test code.

How do you guys do such optimizations? or test matrix management? Is there a better practice, in another testing framework?

Answer 1

Until recently, I've always thought of UI Automation as black box testing where my UI tests drive against a fully stand alone web site or application. As a result, the tests run under the constraint of normal execution and are subject to a host of environment overhead issues.

I've recently adopted the notion of "shallow" and "deep" UI tests where each set of tests run under an optimized configuration to ease environmental differences and speed things up. For example, the login controller is swapped out with a mechanism that avoids OAuth login overhead and is hard coded with fixed usernames. The product catalog skips database lookup and is hard coded with a few fixed items. The ecommerce backend is swapped out to perform speedy operations that accept/reject transactions based on credit card and amount.

Under a "shallow" configuration I can perform "deep" testing against the UI logic. When I switch to a "deep" configuration, it resembles production and I can perform "shallow" testing of fully integrated components such as login, product catalog, search, etc.

A mix of testing strategies is required.

Answer 2

May be the ui-test-automation-best-practices article is helpful for you. It has some examples how to improve performance of automating ui testing by minimizig logins and context changes.