python-ecdsa icon indicating copy to clipboard operation
python-ecdsa copied to clipboard

[WIP] Add mutation testing to CI

Open tomato42 opened this issue 4 years ago • 5 comments

related to #131

0.14.1: With just 20 mutants executed in 25 minutes, it's not really useful (the confidence interval is way too large) 0.15-beta (8deb089e7d): Single test suite execution takes about 9.97s on Travis, in 25 minutes we execute 107 tests, so better, but that still translates to a 99.9% confidence interval of 5.39% to 26.38% for survival rate. So we will need a faster test suite still. 0.16.0: No significant change, about 152 tests in 25 minutes,

todo:

  • [x] implement @slow decorator (pytest docs)
  • [x] change hypothesis settings when option to run slow tests is enabled to execute fewer examples
  • [ ] mark all the slow tests, including hypothesis tests, that take more than few miliseconds to execute, with the @slow decorator
  • [ ] create new static test cases so that branch coverage doesn't fall with the slow tests disabled
  • [ ] use small curve (#24, #223), use it, as it's faster, to test correctness of the implementation instead of P-256, P-224 or P-192

tomato42 avatar Nov 02 '19 18:11 tomato42

Coverage Status

Coverage increased (+0.1%) to 98.699% when pulling 2273354d19e8c50b2a4c0bd673c5ccb71c7f52a1 on tomato42:cosmic-ray into bbe36794ae60683b1095098893570360f86c7084 on warner:master.

coveralls avatar Nov 02 '19 18:11 coveralls

so, with those changes we end up with 222 test suites executed (6.7s per test suite execution), better, but not really good, that still means a 99.9% confidence interval of 10 percentage points

looks like we need to get down to around 1s per test suite execution to get workable confidence intervals

tomato42 avatar Nov 12 '20 12:11 tomato42

CI run: https://travis-ci.org/github/warner/python-ecdsa/builds/747963586

tomato42 avatar Dec 06 '20 17:12 tomato42

CI run: https://travis-ci.org/github/warner/python-ecdsa/builds/747987147

tomato42 avatar Dec 06 '20 19:12 tomato42

CI run: https://travis-ci.org/github/warner/python-ecdsa/builds/748034897

tomato42 avatar Dec 07 '20 02:12 tomato42