python-ecdsa
python-ecdsa copied to clipboard
[WIP] Add mutation testing to CI
related to #131
0.14.1: With just 20 mutants executed in 25 minutes, it's not really useful (the confidence interval is way too large) 0.15-beta (8deb089e7d): Single test suite execution takes about 9.97s on Travis, in 25 minutes we execute 107 tests, so better, but that still translates to a 99.9% confidence interval of 5.39% to 26.38% for survival rate. So we will need a faster test suite still. 0.16.0: No significant change, about 152 tests in 25 minutes,
todo:
- [x] implement
@slow
decorator (pytest docs) - [x] change hypothesis settings when option to run slow tests is enabled to execute fewer examples
- [ ] mark all the slow tests, including hypothesis tests, that take more than few miliseconds to execute, with the
@slow
decorator - [ ] create new static test cases so that branch coverage doesn't fall with the slow tests disabled
- [ ] use small curve (#24, #223), use it, as it's faster, to test correctness of the implementation instead of P-256, P-224 or P-192
Coverage increased (+0.1%) to 98.699% when pulling 2273354d19e8c50b2a4c0bd673c5ccb71c7f52a1 on tomato42:cosmic-ray into bbe36794ae60683b1095098893570360f86c7084 on warner:master.
so, with those changes we end up with 222 test suites executed (6.7s per test suite execution), better, but not really good, that still means a 99.9% confidence interval of 10 percentage points
looks like we need to get down to around 1s per test suite execution to get workable confidence intervals
CI run: https://travis-ci.org/github/warner/python-ecdsa/builds/747963586
CI run: https://travis-ci.org/github/warner/python-ecdsa/builds/747987147
CI run: https://travis-ci.org/github/warner/python-ecdsa/builds/748034897