Python nose critical vs non-critical tests

Question

I am still learning testing in Python and cannot find a straight forward answer to this question.

I am using python to drive selenium testing on my application. I run the test-suite with Nose.

Some tests are critical and must pass in order for code check-ins to be acceptable. Others tests will sometimes fail due to factors outside of our control and are not critical.

Is there a standard Nose Plugin that would allow me to specify which test are not critical and give me a report with that break down? Or is there some other standard way of doing this?

Answer 1

You can always use attrib plugin and decorate your critical tests with @attr('critical')

Within your CI, run nose twice, once with -a critical=True (and condition your checkin/deployment on that) and -a critical=False.

Answer 2

First of all, it's probably a bad idea at all to have tests that "are allowed to fail because they are not critical". You should try to mitigate the influence of external factors as much as possible, and have an environment that allows you to consistently run your tests and especially reproduce any errors you may find.

That said, reality can differ a lot from from theory, so here we go:

Is there a standard Nose Plugin that would allow me to specify which test are not critical and give me a report with that break down?

No, at list not among the built-in plugins or the third-party plugins mentioned on their website.

Or is there some other standard way of doing this?

For pyunit (and consequently nose), a test can only pass or fail; there is nothing in between. To get a better overview when looking at test results, I would keep such tests in a separate test suite, independent from the regular "must-pass" tests. Furthermore, if these unimportant tests are allowed to fail without blocking the check-in, it sounds fair to me that their execution should also be made optional.