Automated JavaScript testing with Intern fails randomly

Question

We are using Intern to run a suite of functional JavaScript tests for our application, against an Ubuntu VM launched using Vagrant.

Each test bootstraps itself with a data file that generates some UI output. Each of the tests run successfully in isolation. In fact, in many cases the tests run successfully together too.

However, we occasionally see failures of a selection of the tests. Currently we have 16 tests which are being run against Chrome and Firefox. It is not unusual to see a list of failures of maybe 4 of the 16 tests. If I isolate those tests into a smaller suite and re-run the tool, the tests pass successfully again.

Now, in theory, the tests are individually isolated from one another and the actual number of failures varies each time this is run so I don't think the fault is that one test is affecting another. The server that is providing the platform for the testing is running locally, on a very well resourced machine.

Is there a way to write the tests or a way to run Intern that will give us more reliable results?

Answer 1

We have discovered what was causing our tests to fail randomly when run together as a suite. One of our tests used modifier keys such as [Shift] and [Ctrl]. The limited documentation we have for WebDriver says of the keys command:

Press keys (keys may still be down at the end of command).

It seems that sometimes this command was leaving a modifier key depressed and therefore active for the next test in the chain. As some of our other tests use [Tab] sequences to reach certain page elements, and [Shift][Tab] goes backwards through the page, our tests were failing depending on their position in the suite.

We do still have issues to do with timing but the situation of random failures has been isolated to this problem.