python-ecdsa
python-ecdsa copied to clipboard
Verify quality of test coverage
Use mutation testing to verify if the existing test coverage is good enough.
Options:
- cosmic-ray (docs) (see also https://github.com/sixty-north/cosmic-ray/issues/484)
- mutmut
given that we are able to verify something like 20 mutants in 25 minutes in Travis, we would be able to mutation test only the simplest of simple patches – unusable without #127, maybe usable with it
either way, not for 0.14
side note: now, just before 0.14 is released (d6cb2886c53923), the test suite has 23.93% survival rate (76.07% mutation score)
with 0.16 we improved a bit, the survival rate is 21.33%
with 0.18.0 we improved a lot, the survival rate is now 16.86%, though I executed the unchanged test suite (not the one with changes from https://github.com/tlsfuzzer/python-ecdsa/pull/156) and cosmic-ray changed significantly too
And before 0.19 release the survival rate is 21.51% (mutation score 78.49%), but we're able to execute about 33% of all mutations, so the confidence interval for survival rate is quite narrow: 20.80 to 22.22%.