Getting useful GCov results for header-only libraries

Question

For my header-only C++ library (lots of templates etc) I use GCov to check test coverage. However, it reports 100% coverage for all headers because the unused functions aren't generated by the compiler in the first place. Manually spotting uncovered functions is easy but defeats the purpose of continuous integration…

How does one solve this automatically? Should I just use "lines hit / LOC" as my coverage metric and just never reach 100% again?

Answer 1

Apart from the usual flags to GCC controlling inlining;

--coverage -fno-inline -fno-inline-small-functions -fno-default-inline

You can instantiate your template classes at the top of your unit test files;

template class std::map<std::string, std::string>;

This will generate code for every method in that template class making the coverage tools work perfectly.

Also, make sure that you initialise your *.gcno files (so for lcov)

lcov -c -i -b ${ROOT} -d . -o Coverage.baseline
<run your tests here>
lcov -c -d . -b ${ROOT} -o Coverage.out
lcov -a Coverage.baseline -a Coverage.out -o Coverage.combined
genhtml Coverage.combined -o HTML

Answer 2

I'm also using GCov to check test coverage (Tests written with Google Test framework), additionally I use the Eclipse GCov integration plugin or the LCov tool to generate easy to inspect views of the test coverage results. The raw GCov output is too hard to use :-(.

If you have header only template libraries, you also need to instrument (using G++ flag --coverage) your testing classes that instantiate the template classes and template member functions to see reasonable GCov outputs for these.

With the mentioned tools it's easy to spot template code that wasn't instantiated with the test cases at all, since it has NO annotations.

I have setup a sample and copied the LCov output to a DropBox link you can inspect.

Sample code (TemplateSampleTest.cpp is instrumented using the g++ --coverage option):

TemplateSample.hpp

template<typename T>
class TemplateSample
{

public:
    enum CodePath
    {
        Path1 ,
        Path2 ,
        Path3 ,
    };

    TemplateSample(const T& value)
    : data(value)
    {
    }

    int doSomething(CodePath path)
    {
        switch(path)
        {
        case Path1:
            return 1;
        case Path2:
            return 2;
        case Path3:
            return 3;
        default:
            return 0;
        }

        return -1;
    }

    template<typename U>
    U& returnRefParam(U& refParam)
    {
        instantiatedCode();
        return refParam;
    }

    template<typename U, typename R>
    R doSomethingElse(const U& param)
    {
        return static_cast<R>(data);
    }

private:
    void instantiatedCode()
    {
        int x = 5;
        x = x * 10;
    }

    void neverInstantiatedCode()
    {
        int x = 5;
        x = x * 10;
    }
    T data;
};

TemplateSampleTest.cpp

#include <string>
#include "gtest/gtest.h"
#include "TemplateSample.hpp"

class TemplateSampleTest : public ::testing::Test
{
public:

    TemplateSampleTest()
    : templateSample(5)
    {
    }

protected:
    TemplateSample<int> templateSample;

private:
};

TEST_F(TemplateSampleTest,doSomethingPath1)
{
    EXPECT_EQ(1,templateSample.doSomething(TemplateSample<int>::Path1));
}

TEST_F(TemplateSampleTest,doSomethingPath2)
{
    EXPECT_EQ(2,templateSample.doSomething(TemplateSample<int>::Path2));
}

TEST_F(TemplateSampleTest,returnRefParam)
{
    std::string stringValue = "Hello";
    EXPECT_EQ(stringValue,templateSample.returnRefParam(stringValue));
}

TEST_F(TemplateSampleTest,doSomethingElse)
{
    std::string stringValue = "Hello";
    long value = templateSample.doSomethingElse<std::string,long>(stringValue);
    EXPECT_EQ(5,value);
}

See the code coverage output generated from lcov here:

TemplateSample.hpp coverage

Caveat: 'Functions' statistics is reported as 100%, which is not really true regarding the not instantiated template functions.

Answer 3

Since I found this question super useful in setting up test coverage for my header-only library, here are some additional things I learned in hopes they can help others:

Even with all of the flags mentioned in these answers, I was still having problems with unused class methods getting optimized out. After much experimenting, I found that clang source based coverage (these flags: -fprofile-instr-generate -fcoverage-mapping) includes all class methods and is in general the most reliable way of getting coverage data. I also use the flags: -O0 -fno-inline -fno-elide-constructors to further reduce the risk of code getting optimized out.

For a large library, the template instantiation thing is still a problem. Explicitly instantiating them is all well and good, but if anyone ever forgets to, you'll get inaccurate code coverage metrics. See my answer to this question for an approach to automatically adjusting code coverage data to account for this.

Answer 4

I stumbled across this problem too and unfortunately didn't have much luck with the various flags mentioned, I did, however, discover two ways to generate more accurate coverage information when dealing with header-only functions.

The first is to add the flag -fkeep-inline-functions (https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#index-fkeep-inline-functions).

This did give me just the results I was after but came with some serious problems trying to integrate with other libraries (even the normal C++ standard library). I wound up getting link errors because certain functions that should have been removed by the linker were not (e.g. a function declaration with no definition).

The second approach (the one I opted for in the end) was to use __attribute(used)__ in GCC to annotate all my header API functions. The documentation (https://gcc.gnu.org/onlinedocs/gcc-4.3.0/gcc/Function-Attributes.html) states:

used

This attribute, attached to a function, means that code must be emitted for the function even if it appears that the function is not referenced.

I used a #define to wrap it so I only have it turned on when I'm using GCC and coverage is enabled:

#ifdef _MSC_VER
#define MY_API
#elif defined __GNUC__ && defined COVERAGE
#define MY_API __attribute__((__used__))
#endif // _MSC_VER ? __GNUC__ && COVERAGE

Usage then looks like this:

MY_API void some_inline_function() {}

I'm going to try and write up how I got everything working at some point which I'll link to from here in future if I ever get round to it 😁

(Note: I also used -coverage -g -O0 -fno-inline when compiling)