kcov
kcov copied to clipboard
SimonKagstrom
kcov generates inconsistent output when there are files with duplicate names
I noticed kcov was generating inconsistent outputs for certain files in my project, and after a while I realized it was because the files in question had an identical name to at least one other file in the project.
We were putting platform-specific implementations in sub-folders with the same name (e.g. SomeObject.cpp in /src, with some OSX-specific implementations under /src/osx/SomeObject.cpp).
Despite the different filepaths, code coverage reporting for those files was slightly different after each execution, both in terms of instrumented lines and code covered, making it impossible to compare code coverage from one run to the next.
Obvious workaround is to make sure all file names are unique throughout the project, but the problem is likely to spring up again and again as the project expands.
Here's some example outputs for one such file after 5 runs to give you an idea of what you can expect:
Date: 2017-06-08 14:12:21 Instrumented lines: 74 Code covered: 74.3% Executed lines: 55
Date: 2017-06-08 14:12:31 Instrumented lines: 71 Code covered: 73.2% Executed lines: 52
Date: 2017-06-08 14:12:42 Instrumented lines: 74 Code covered: 74.3% Executed lines: 55
Date: 2017-06-08 14:12:51 Instrumented lines: 72 Code covered: 76.4% Executed lines: 55
Date: 2017-06-08 14:13:01 Instrumented lines: 74 Code covered: 74.3% Executed lines: 55
Is this only for the summaries, or are the actual file contents wrong as well?
kcov should handle this case, I had issues with non-unique filenames early on, although that was with output filename collisions.
It seems to be both the file contents and the summary (coverage.json) are affected.
Here's an example diff of the coverage.json files (names replaced to protect the accused). The total lines, percent covered and even which files even appear seems to vary between one run and the next.
The root of the issue is that there are two files named FileC.cpp (/FileC.cpp and /posix/FileC.cpp) and those files call into functions used by the others seen here, so they just seem to getting dragged along for the ride due to the naming conflict of FileC.
diff kcov-output-1/run-tests/coverage.json kcov-output-2/run-tests/coverage.json
3a4
> {"file": "<some source path>/apple/FileA.mm", "percent_covered": "0.00", "covered_lines": "0", "total_lines": "9"},
6a8,9
> {"file": "<some source path>/posix/FileB.h", "percent_covered": "0.00", "covered_lines": "0", "total_lines": "1"},
> {"file": "<some source path>/FileC.cpp", "percent_covered": "16.67", "covered_lines": "11", "total_lines": "66"},
22,25d24
< {"file": "<some source path>/posix/FileB.h", "percent_covered": "0.00", "covered_lines": "0", "total_lines": "1"},
< {"file": "<some source path>/FileC.h", "percent_covered": "0.00", "covered_lines": "0", "total_lines": "1"},
< {"file": "<some source path>/apple/FileA.mm", "percent_covered": "0.00", "covered_lines": "0", "total_lines": "9"},
< {"file": "<some source path>/posix/FileD.cpp", "percent_covered": "0.00", "covered_lines": "0", "total_lines": "14"},
50c49
< {"file": "<some source path>/FileE.h", "percent_covered": "28.57", "covered_lines": "4", "total_lines": "14"},
---
> {"file": "<some source path>/FileE.h", "percent_covered": "58.33", "covered_lines": "14", "total_lines": "24"},
59a59
> {"file": "<some source path>/FileD.cpp", "percent_covered": "0.00", "covered_lines": "0", "total_lines": "4"},
68d67
< {"file": "<some source path>/FileD.h", "percent_covered": "0.00", "covered_lines": "0", "total_lines": "1"},
76d74
< {"file": "<some source path>/posix/FileC.cpp", "percent_covered": "0.00", "covered_lines": "0", "total_lines": "10"},
82c80
< {"file": "<some source path>/FileF.h", "percent_covered": "36.36", "covered_lines": "4", "total_lines": "11"},
---
> {"file": "<some source path>/FileF.h", "percent_covered": "30.77", "covered_lines": "4", "total_lines": "13"},
88c86
< {"file": "<some source path>/FileG.h", "percent_covered": "2.80", "covered_lines": "3", "total_lines": "107"}
---
> {"file": "<some source path>/FileG.h", "percent_covered": "8.41", "covered_lines": "9", "total_lines": "107"}
90,92c88,90
< "percent_covered": "14.05",
< "covered_lines": 477,
< "total_lines": 3394,
---
> "percent_covered": "14.61",
> "covered_lines": 504,
> "total_lines": 3450,
96c94
< "date": "2017-06-09 10:58:43"
---
> "date": "2017-06-09 10:58:54"
Sorry, didn't realise this was on OSX. I'll try to reproduce it there (it was OK on Linux at least).
It will probably be after the weekend though. Thanks for the detailed report!
Are these subsequent runs on the same output directory, or is the output directory re-created each time? I.e, do you do
kcov /tmp/kcov <program>
kcov /tmp/kcov <program-again>
or
kcov /tmp/kcov <program>
rm -rf /tmp/kcov
kcov /tmp/kcov <program>
? Also, are these subsequent runs of the same executable, or have you changed the source code and recompiled in between. On thing which is lacking on the OSX port is checksum support for the binaries, so kcov won't detect if the program has been recompiled since last run. That would lead to inconsistencies if the same output directory is re-used. With checksumming, kcov would just start from scratch again.
I've tried on my OSX machine, but at least with a trivial test program, same-file-name-in-multiple-directories work for me.
Its the former. Same executable, no changes, different output directories each time, and I totally erased all existing output folders between test batches for sanity, because I suspected something like that might be the case.
Wild stab in the dark, but it might be because the class definition is incomplete without the object files being linked together? The file in the base folder is the general implementation for a given class, while the duplicate files in subfolders contain platform-specific implementations of certain member functions.
OK, thanks for the test. Then it's not the accumulation that got it wrong at least.
I don't think the source layout should really matter, but I might be wrong. If that would be the case, i.e., that the debug information is generated incorrectly, similar problems should occur in lldb itself as well. But then it should be the same between runs, so I don't think that is the issue.
Should the program behave exactly the same between runs? I notice you sometimes have 55 lines executed and sometimes 52. Is that expected or also a part of the problem?
Once I corrected the issue, by making sure all files have unique names (e.g. FileD.cpp and posix/FileD_Posix.cpp), then the output was consistent every time afterwards, so I'm confident the program and the debugging stuff works in the absence of the issue.
Yeah, that inconsistency is the problem. The number of instrumented and executed lines changes from one run to the next.
At the very least I have a workable workaround. Just need to make sure nobody submits files with duplicate names in the future, which should be easy enough with a bash command in a jenkins task.
One thing you could try is to run with
kcov --debug=1 <other opts>
You should then see a lot of lines with file:line -> address mappings such as:
REPORT .../main.cc:9 at 0x9e2a5d1c1
and perhaps grepping for your FileD.cpp during subsequent runs could give some clue to what goes wrong. They should really be the same each time.
I don't know if I should create a new issue for this, but it seems like there's still a bug when running scripts with the same filename -- kcov doesn't automatically merge their output until you run something with a different filename.
See example here
I think this is probably a separate issue, at least technically, since it's using the bash "engine" and the merge-mode. So probably a new issue would be good to have to track this.