BetterReflection icon indicating copy to clipboard operation
BetterReflection copied to clipboard

Infection: raise timeout for mutants from 10 to 60 seconds

Open Slamdunk opened this issue 2 years ago • 3 comments

The current 10 seconds timeout hides mutants that in reality are escaping if we raise it to just 60 seconds:

3676 mutations were generated:
    3607 mutants were killed
       0 mutants were configured to be ignored
       3 mutants were not covered by tests
      38 covered mutants were not detected
      14 errors were encountered
       0 syntax errors were encountered
      14 time outs were encountered
       0 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 98%
         Mutation Code Coverage: 99%
         Covered Code MSI: 98%

/cc @kukulich

Slamdunk avatar Feb 13 '23 07:02 Slamdunk

The mutation score is really like some neverending zombie TV serie :)

kukulich avatar Feb 13 '23 08:02 kukulich

Github Actions is freaking slow, 60 seconds are not enough for another 392 mutants, while on my i7-8700 even with 30 seconds no mutants time out:

https://github.com/Roave/BetterReflection/actions/runs/4161271803/jobs/7199509066#step:5:116

3676 mutations were generated:
    3242 mutants were killed
       0 mutants were configured to be ignored
       7 mutants were not covered by tests
      15 covered mutants were not detected
      11 errors were encountered
       0 syntax errors were encountered
       9 time outs were encountered
     392 mutants required more time than configured

I'll leave up to you to tweak it: if you raise it even higher you'll eventually get timed out mutants to zero, but the build could take almost 2 hours :roll_eyes:

Slamdunk avatar Feb 13 '23 09:02 Slamdunk

@Slamdunk thanks for investigating this meanwhile! :muscle:

Ocramius avatar Feb 13 '23 16:02 Ocramius