How can I get code coverage data from Python BDD functional tests using Behave?

Question

I haven't seen an answer for this specific question (Test coverage tool for Behave test framework) and I haven't seen any Google search results produce a sufficient answer. Therefore...

How can I get a code coverage report from Behave? I find it hard to believe that there are no Python developers using BDD methodology and I find it even harder to believe that those Python developers who are using BDD are doing so without code coverage statistics from their functional tests. Can Coverage.py be used to drive Behave to produce code coverage? How?

Answer 1

I don't know how to use behave, but I used Cucumber for BDD, which I think probably almost similar. And so I think you should be able to use behave with coverage. you have to specify which file to include in the file.. (I used it with cucumber). See if this might help.

Hope this answer your question :)

# .coveragerc to control coverage.py
[run]
parallel = True

# if you want to include tests append tests/*
include =
    src/*
    *src*

[paths]
source =
    src/
    */src

tests =
    tests/
    */tests

Answer 2

Following on from David's suggestion above.

Assuming the code to be tested is in an app directory, add the following to your .coveragerc file:

[run]
source=app/

From the terminal:

coverage run $(which behave);

You can then use coverage report or coverage html as normal. If you don't specify the app dir in your .coveragerc file, coverage will test all the Python libraries local to your behave installation.

Answer 3

Behave can generate junit coverage data and the coverage package can combine this data from multiple test runs as well as produce an HTML report that you can peruse or automatically publish in your CI environment.

Here are the statements I currently use to produce, combine, and report on coverage with behave:

cd your/repo/tests  # <-- Make sure you're inside your tests dir!
rm -rf behave-reports/*
behave --junit --junit-directory behave-reports
coverage combine
coverage html

The rm -rf behave-reports/* forcefully removes everything inside the behave-reports/ directory so that I am guaranteed either a fresh coverage report or nothing at all (producing a failure in CI, in my case). Note that if you run your tests locally you'll want to add an entry to your .gitignore file (or equivalent) so that you aren't adding and committing your test results.

Running behave with --junit will produce junit output, while the --junit-directory flag tells behave where to write that junit data on disk.

Running coverage combine eats all of the code coverage and combines that into a single coverage file.

Finally, coverage html produces a pretty html report that includes all of the combined coverage data.

Answer 4

Another option, use coverage to run behave's main script:

coverage run /path/to/lib/python2.7/site-packages/behave/__main__.py

Of course you'll want to specify in your .coveragerc which source files you want to include.