fff
fff copied to clipboard
fff needs code coverage
With the ever increasing PRs and additional features, fff could do with some code coverage stats.
This is a perfect write up on gcov for TravisCI and OpenCppCoverage for AppVeyor.
https://codingnest.com/how-to-get-code-coverage-from-ci/
Sounds like a great idea....although I wonder how good the coverage data will be? If we don't use a pre-processor macro it's code definition will not end up in a translation unit. So maybe we need to force instantiate the macros somehow?
Isn't that what the tests do?
On Thu., 6 Dec. 2018, 4:31 pm Mike Long <[email protected] wrote:
Sounds like a great idea....although I wonder how good the coverage data will be? If we don't use a pre-processor macro it's code definition will not end up in a translation unit. So maybe we need to force instantiate the macros somehow?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/meekrosoft/fff/issues/68#issuecomment-444753448, or mute the thread https://github.com/notifications/unsubscribe-auth/AA6TtGYKqiHgcRwLsnX7awJcUK6T5Hk9ks5u2Ku2gaJpZM4ZFJBY .
Sorry, I'm not being very clear :-D What I mean is, there isn't really any executable code in fff...it's just a header file that contains macros which may instantiate data structures. So I'm struggling to see what code coverage can help with in this case?
Yes, you're correct. The preprocessor gets in the way of coverage testing. I'll need to rethink this.
Might still be useful to ensure the tests actually get run though?
@meekrosoft, I had a little look at this just now, a cursory idea; haven't thought hard enough about reasons this might not be useful:
gcc -E fff.h > out1.txt
gcc -E test/fff_test_c.c > out2.txt
awk 'a[$0]++' out1.txt out2.txt | wc -l # 2834
cat out1.txt | wc -l # 327
cat out2.txt | wc -l # 3898
# total lines: 327+3898=4225
# percentage of match: 2834/4225*100=~67%
With this method, we could calculate an average across all test files and use it as a heuristic.
NOTE: the above method can't work given that the pre-processed c file, e.g. test/fff_test_c.c
includes all of fff.h
.
Another approach that I'm playing with (haven't a good result yet):
#!/usr/bin/env bash
fff_len=`cat fff.h | wc -l`
# test/fff_test_c.c
gcc -E test/fff_test_c.c > out
comm -13 <(sort < fff.h | uniq) <(sort < out | uniq) > unique_to_out # strip out fff.h include
awk 'FNR==NR{a[$1];next}($1 in a){print}' unique_to_out fff.h > matched # find unique matches
grep -Fxf <(sort < matched | uniq) <(sort < fff.h | uniq) > coverage # find coverage of matches in fff.h
cov_len=`cat coverage | wc -l`
echo "test/fff_test_c.c coverage:"
echo "$cov_len / $fff_len" | bc -l
# test/fff_test_cpp.cpp
gcc -E test/fff_test_cpp.cpp -I . > out
comm -13 <(sort < fff.h | uniq) <(sort < out | uniq) > unique_to_out
awk 'FNR==NR{a[$1];next}($1 in a){print}' unique_to_out fff.h > matched
grep -Fxf <(sort < matched | uniq) <(sort < fff.h | uniq) > coverage
cov_len_2=`cat coverage | wc -l`
echo "test/fff_test_cpp.cpp coverage:"
echo "$cov_len_2 / $fff_len" | bc -l
# total coverage
echo "TOTAL COVERAGE:"
echo "($cov_len + $cov_len_2) / ($fff_len * 2)" | bc -l
Mmm, interesting approach....