stryker-net icon indicating copy to clipboard operation
stryker-net copied to clipboard

Published 20 hours ago • verified publisher icon stryker-mutator

Solution runs are inconsistant

Open richardwerkman opened this issue 4 years ago • 2 comments

Describe the bug I have seen a project getting different stryker results every time I ran it. It happens when running the full solution.

Buildalyzer seemed to find less test projects than expected on some runs.

Expected behavior I expect a consistant result every run.

Desktop (please complete the following information):

  • OS: Windows
  • Type of project core 3.1
  • Framework Version NET 5.0
  • Stryker Version 0.21.0

Additional context The solution run feature is still experimental.

richardwerkman avatar Jan 23 '21 10:01 richardwerkman

I have similar results of different scores when running full solution with multiple test projects. I have attached the logs. log-score-29.51.txt log-score-42.08.txt

1itachi avatar Jan 29 '21 18:01 1itachi

I quite consistently observed 4 different results when using solution runs on 0.21.1 Of course this may apply only for my particular solution (4 projects, 3 test projects), but what it usually happens is either:

  1. No mutatations produced at all (and no report generated)
  2. Just 0 mutants killed
  3. Result that is roughly at 50% of expected mutations score
  4. Result that is kinda close to running all projects separately. I didn't compare it very thoroughly though.

wiciok avatar Mar 06 '21 01:03 wiciok