quadiron
quadiron copied to clipboard
Add code coverage into the CI
It would be really nice to compute the code coverage in our CI pipeline, especially since we can upload the result to CircleCI's artifacts directory (BTW, we should propably do the same for our benchmark).
That being said, before getting there we have several changes to make if we want to do it.
Add coverage tooling (easy)
We should install a recent lcov
in our Docker image for testing.
Note that it must be a recent lcov
because older version don't work with GCC 8.
Make the test suite faster
In order to have a comprehensive coverage, we should run the test suite 3 times:
- once without SIMD
- once with SSE
- once with AVX
Otherwise, some code path won't be covered by the test suite.
In order to keep a bearable user experience, we should remove the stuff that unecessarily slow don our test suite.
First, we should get rid of test_ec.sh
:
- it's way too slow (you can't run the test suite in Debug mode, and I'm not sure Release mode give an accurate coverage)
- it's too coarse grained
test_ec.sh
is based on ec_test
, which could be good for integration test but not for unit testing.
I think we should convert as much as possible the test from test_ec.sh
into unit tests, and for the few thing that do require a full run then we can add some integration test.
Next, we should remove most of the loop that run the tests thousands of time with random input:
- that make the tests non-reproducible
- that's not the role of unit test: if we want to test our behavior with random input to detect numeric instabilities or other bugs this should be done by using instrumentation-guided fuzzing (such as AFL), not by randomly adding randomness hither and yon.
Finally, we should move the benchmark into it's own CircleCI job and update the testing job to run the testing suite thrice before collecting the code coverage results.