c-cube
Coverage with bisect_ppx
As we are adding tests thanks to @jmid, we could add bisect_ppx to produce a coverage report.
Getting an idea of our coverage would be nice!
A few thoughts:
- Have you tried running #206 to see how bad it is? 😅
- I was wondering how we could bring it actively into play - rather than just carry the instrumentation lines around:
- a CI-badge reporting the percentage in the README?
- a CI alarm that triggers if a PR decreases coverage? (could be useful to catch PRs without tests - but potentially annoying...)
- ...
- Finally, if we're going to measure how well our testsuite is doing it would be nice to rebase and squash the test PRs from the fall (sans the shrink logging for a start). I'll see if I can find the time...
Have you tried running Coverage report with bisect_ppx #206 to see how bad it is? sweat_smile
It's not that bad! A checkout to a commit 6 months ago would show a way funnier percentage :).
✦8 Ψ bisect-ppx-report summary --per-file
0.00 % 0/0 src/QCheck_runner.ml
78.57 % 22/28 src/alcotest/QCheck_alcotest.ml
63.93 % 927/1450 src/core/QCheck.ml
67.21 % 859/1278 src/core/QCheck2.ml
12.50 % 20/160 src/ounit/QCheck_ounit.ml
100.00 % 2/2 src/ppx_deriving_qcheck/QCheck_generators.ml
87.50 % 7/8 src/ppx_deriving_qcheck/args.ml
88.89 % 8/9 src/ppx_deriving_qcheck/attributes.ml
96.33 % 236/245 src/ppx_deriving_qcheck/ppx_deriving_qcheck.ml
100.00 % 53/53 src/ppx_deriving_qcheck/tuple.ml
64.87 % 181/279 src/runner/QCheck_base_runner.ml
65.92 % 2315/3512 Project coverage
I was wondering how we could bring it actively into play - rather than just carry the instrumentation lines around:
We can have a badge in the README yes, we also can have a diff percentage in each PR. I also think an alarm when its decreasing will be annoying, the only alarm I'd think would be useful is to detect loss covered lines, but that's practically impossible to do.