Doctest failed with zero exit code

Question

In my test code, my doctest fails but the script exits with a zero return value, which causes the CI run to pass, which is not intended.

Is this the correct behavior of doctest module?

My script ends with:

if __name__ == '__main__':
    import doctest
    doctest.testmod()

The output is like:

**********************************************************************
File "test/test.py", line 7, in __main__
Failed example:
    f(1,0)
Expected:
    -----
    type: <type 'exceptions.ZeroDivisionError'>
    value: integer division or modulo by zero
    x
    -----
Got:
    -----
    type: <type 'exceptions.ZeroDivisionError'>
    value: integer division or modulo by zero
    -----
**********************************************************************
1 items had failures:
   1 of   1 in __main__
***Test Failed*** 1 failures.
tux@iPad:~/programming/exception-notifier(fix-travis)(0)$ echo $?
0

Answer 1

I find using doctest.testmod(raise_on_error=True) will cause an exception to be raised when a test fails, which causes the script exits with a non-zero code.

Python doc here:

Optional argument raise_on_error defaults to false. If true, an exception is raised upon the first failure or unexpected exception in an example. This allows failures to be post-mortem debugged. Default behavior is to continue running examples.

Answer 2

@fossilet's answer works for properly breaking builds that fail tests, but it raises the exception before doctest is able to write anything to the console. This makes your logs much less useful for identifying the problem.

An alternative is to call

sys.exit(doctest.testmod()[0])

This makes the process exit code equal to the number of tests that failed. Your CI tool should interpret nonzero exit codes as failing builds. But the doctest output will still make it to the console.