python-ecdsa icon indicating copy to clipboard operation
python-ecdsa copied to clipboard

Verify quality of test coverage

Open tomato42 opened this issue 4 years ago • 4 comments

Use mutation testing to verify if the existing test coverage is good enough.

Options:

tomato42 avatar Oct 07 '19 09:10 tomato42

given that we are able to verify something like 20 mutants in 25 minutes in Travis, we would be able to mutation test only the simplest of simple patches – unusable without #127, maybe usable with it

either way, not for 0.14

side note: now, just before 0.14 is released (d6cb2886c53923), the test suite has 23.93% survival rate (76.07% mutation score)

tomato42 avatar Nov 03 '19 00:11 tomato42

with 0.16 we improved a bit, the survival rate is 21.33%

tomato42 avatar Nov 12 '20 11:11 tomato42

with 0.18.0 we improved a lot, the survival rate is now 16.86%, though I executed the unchanged test suite (not the one with changes from https://github.com/tlsfuzzer/python-ecdsa/pull/156) and cosmic-ray changed significantly too

tomato42 avatar Jul 09 '22 23:07 tomato42

And before 0.19 release the survival rate is 21.51% (mutation score 78.49%), but we're able to execute about 33% of all mutations, so the confidence interval for survival rate is quite narrow: 20.80 to 22.22%.

tomato42 avatar Jan 15 '24 22:01 tomato42