Why Stryker is so slow?

[caedes]

Question

Why Stryker is so slow running on a create-react-app, Jest, testing-library project?

Stryker environment

We only use the Jest runner plugin.

"@stryker-mutator/core": "^5.5.1",
"@stryker-mutator/jest-runner": "^5.5.1",

With this config file:

{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "_comment": "This config was generated using 'stryker init'. Please see the guide for more information: https://stryker-mutator.io/docs/stryker-js/guides/react",
  "testRunner": "jest",
  "reporters": ["progress", "clear-text", "html"],
  "coverageAnalysis": "off",
  "jest": {
    "projectType": "create-react-app"
  }
}

Additional context

While running Stryker on our 942 unit tests in 73s project covered at 98%, it founds 257 of 1306 files to be mutated and so tries to instrumente 257 source files with 5204 mutants. Estimated time to mutation testing >72h… We never could finish it once. For now, Stryker seems useless for an entire project, we must have missed something.

We run Stryker on some MacBook Pro 2.6 GHz i7 6 cores, 16 Go DDR4.

942 unit tests is a medium project for us, we have hundreds of other projects that can be bigger in our company.

What have we done wrong?

[danny12321]

Hi caedes, is there a specific reason why coverageAnalysis is turned off in your configuration? By default, it is set on perTest so it only runs the test that covers the mutant. This will make it a lot faster.

[caedes]

I tried with "coverageAnalysis": "perTest". Running in progress:

(elapsed: ~35m, remaining: ~19h 39m) 150/5111

I'll wait for a more convincing result, but for now still not runnable in any pipeline.

[nicojs]

You're not doing anything wrong. Stryker is slow because jest is slow (I think). How long does it take to run all tests with Jest?

Did you try to focus on one file for starters? With -m src/file/to/mutate.jsx?

[danny12321]

@nicojs in the first comment @caedes said While running Stryker on our 942 unit tests in 73s.... So I guess you are right that the tests/jest are slow and thereby making Stryker slow.

[nicojs]

Ah yes. Is that 73s with or without jest concurrency?

73s/mutant * 7304 mutants = 148 hours So 72 hours might be expected if 73s is with concurrency. Running Stryker with the jest runner makes it run with concurrency 1, since Stryker is managing concurrent workers.

We have some performance improvements planned for jest. Namely #3236.

We're also currently using the runCLI api, which is a high-level jest API. We might be able to improve performance by using a lower level API, but that currently doesn't exist. Ideas are welcome

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.