Does coverage.py show test coverage on python bindings properly?

Question

I'm interested in test coverage of my functional tests. These tests are testing module code which was compiled via SWIG.

I've tried run:

coverage run --source=/path/to/module/__init__py allMyTests.py

and

coverage run --source=/path/to/module/__init__py partOfTests.py

In every case it saved the same information even though the number of tests and amount of code they are using from module is way different. It does not seem working as I expected but maybe I'm doing something wrong.

Can I find out what is the test coverage of python bindings code at all?

It does not have to be with coverage.py tool.

Answer 1

Coverage is used to determine what proportion of your code is being tested. So two test files that are completely different in amount of code and number of tests may still lead to the same coverage data if they are testing the same parts of your API.

Update:

it sounds like you wanted to test coverage of the C/C++ code in your python extension DLL. Since the extension is C/C++, the coverage module cannot go inside it. What the module can tell you is whether each Python function in your extension gets called from somewhere in your tests.

That being said, any C/C++ code coverage tool that can handle dynamically linked DLL loaded on demand will be able to give you want you want. Ultimately, your script is run by python.exe which eventually loads your DLL. If the DLL is writen say in Visual C++ 2010, and you can find a C/C++ code coverage tool that works with that compiler, then you can tell Visual Studio to attach to the python process, and provide coverage stats for the DLL of interest (I've never done that but I presume that there is a way of telling the coverage tool which portions of the exe -- your DLL in this case -- is of interest).

So I would dig a bit into finding a suitable C/C++ code coverage tool for your C++ compiler. Like on windows this SO post Code coverage in Visual Studio 2010 for native C++ exe/dll indicates there is such tool.