Question: How to merge coverage reports from different binaries

[rianquinn]

I am working on the following project: https://github.com/Bareflank/bsl

We have good line coverage, but branch coverage is the issue. Specifically with any template code. All of the code supports "constexpr" as it is all C++20, so all of the unit tests are executed at both compile-time and runtime. Since undefined behavior is not allowed in a constexpr, we use the compile-time version of each unit test to ensure UB never happens, and then we use the runtime version for coverage data. Compile-time tests increase the time it takes to compile the code, we we broke the unit tests into different binaries. For example, when compiling our safe_integral (an integer class that tracks the carry bit), we have a different test for each integral type (like uint8_t, uint16_6, uint32_t, etc...).

The unit test logic itself however uses safe_integrals, and so in the test for uint8_t for example, the code might also touch some of the int32_t versions of this class. The build system currently uses grcov, but we are looking for a better option. Right now, when using gcovr (i.e. this project), when you run tool, it collects all of the tests from all of the binaries. For line coverage, this works great. But for branch coverage, we get strange issues with branch coverage.

A single if statement might have 20 different "branches". I assumed at first that this was due to templates, and it is, but it is more than that. When looking at the gcov results, I can see the uint8_t that I am testing, and it hits 100% of all branches. Because the int32_t is used in the same code just to handle some misc stuff, I see a bunch of misses for the int32_t version. In another test however, we test the in32_t, and it hits 100% of all branches.

The issue is, these two tests are not being merged. The branch coverage shows fails for int32_t in the uint8_t (and all other vesions except for the in32_t version of the test). So if I test 8, 16, 32 and 64 for signed and unsigned, I don't get 8 versions of the branch, I see 16 because the in32_t version shows up in every single test, and in all of the tests, it shows that most of the branches are not covered, even though in one of the tests we hit 100%.

How do I merge these? If I hit 100% in one binary for int32_t, any other binaries that also use in32_t should not add additional branches that state they fail so long as the type is the same.

[latk]

Since branches have no stable identifiers, it is not generally possible to merge branch coverage data across multiple test runs if templates are involved. The alternative would be to ensure that all instantiations are tested as part of the same executable.

This is a limitation in gcov which gcovr cannot work around. Grcov might be able to work around this since it parses the binary LLVM coverage data format, not sure how it handles this. Presumably, these lower-level tools can recognize that branches belong to different template instantiations of a function, and can thus account for them separately. But again: not possible for gcovr since it never touches the raw gcda counters and has no concept of functions.

In my experience with C++, it's difficult to get useful information from branch coverage information. The resulting reports could be useful to guide testing effort, but are probably not useful for quality-gating.

To clarify: I assume that you're trying to merge coverage reports using gcovr --add-tracefile from multiple executables that were compiled separately. Merging multiple runs of the same executable should work just fine, as should collection of data from a single run of a single executable that consists of multiple compilation units. If those were to produce inconsistent results, that would be a bug.