Stryker.Js never find tests for mutation

[AndreLZGava]

Question

I'm trying to configure stryker in our project, its really big, I'd like to configure it for react components and for native Js functions, for now, I'm trying to configure it for react, in react we use jest for test, I've set my stryker.conf.json with this structure:

{
  "ignorePatterns": [
    'manyDirsFromProject'
  ],
  "mutate": [
    "all/path/untill/get/original/component/file.js",
  ],
  "testRunner": "jest",
  "reporters": [
    "html"
  ],
  "coverageAnalysis": "perTest",
  "jest": {
    "projectType": "custom",
    "enableFindRelatedTests": true,
    "configFile": "jest.config.js",
    "config": {
      "testMatch": ["<rootDir>/all/path/untill/get/original/component/file/**/*.test.js"],
      "testEnvironment": "jest-environment-jsdom-sixteen"
    }
  },
  "cleanTempDir": true,
  "eventReporter": {
    "baseDir": "stryker"
  },
  "htmlReporter": {
    "fileName": "stryker/mutation.html"
  },
  "packageManager": "yarn",
  "tempDirName": "stryker-tmp",
  "ignoreStatic": true,
  "incremental": true,
  "incrementalFile": "stryker-tmp/incremental.json",
  "allowEmpty_comment": true
}

I turn allowEmpty to comment cause by running with it enabled I got this error: (28) WARN OptionsValidator Unknown stryker config option "allowEmpty" (Its not the big problem just told bout it).

And matter what I did every time I run the comand stryke run this returns: (28) ERROR Stryker No tests were executed. Stryker will exit prematurely. Please check your configuration.

I already done it trying to get log trace but everything seems fine, but it cannot find tests (tests for components are always in the same dir/tests/component.test.js)

our jest.config.js is like this:

module.exports = {
  testEnvironment: 'jest-environment-jsdom',
  testMatch: [
    '<rootDir>/all/path/**/*.test.js'
  ],
  modulePaths: ['<rootDir>/all/path/'],
  modulePathIgnorePatterns: [
  '...'
  ],
  moduleNameMapper: {
   'allModules'
  },
  coverageDirectory: '<rootDir>/react-coverage',
  coverageReporters: [['lcov', { file: 'lcov-react.info' }]],
  setupFiles: ['<rootDir>/setup-jsdom-test.js', 'raf/polyfill'],
  setupFilesAfterEnv: ['<rootDir>/setup.react.js'],
  testEnvironmentOptions: {
    url: 'http://localhost'
  },
  transformIgnorePatterns: ['node_modules/?!(garden)'],
  snapshotSerializers: ['@emotion/jest/serializer']
}

When I ran tests for components all of them run and works fine, but Stryker cannot find them. Does anyone know what do I could do to make Stryke get and run related tests?

Stryker environment

 "@stryker-mutator/core": "^6.4.2",
 "@stryker-mutator/jest-runner": "^6.4.2",

Additional context

I'm running it on docker with yarn 1.22.19

[nicojs]

Hi @AndreLZGava, thanks for opening this issue and the detailed description ♥

What happens if you set jest.enableFindRelatedTests to false? Maybe something in your setup prevents jest --findRelatedTests from working.

I've also noticed that you're overriding the "testEnvironment" with "jest-environment-jsdom-sixteen", but you're running with "jest-environment-jsdom" normally. You should be able to simply remove "testEnvironment" from your stryker.conf.json file, it is not needed.

[AndreLZGava]

Hi @nicojs

thanks for your reply, I put this 2 options in the last tryes I did, but I decided to remove testEnvironment as you said and turn jest.enableFindRelatedTests to false, I ran the command and I got the same result

(28) ERROR Stryker No tests were executed. Stryker will exit prematurely. Please check your configuration.
error Command failed with exit code 1.

Do you have any other idea that I could try?

[nicojs]

Hmm interesting. What does your normal jest command look like for running your tests? Is it simply jest? And do you run that from the same directory as you run Stryker?

[AndreLZGava]

Actually I ran it from docker but I can enter into container as shell and run from there, my command to run test is:

yarn jest --config=jest.config.js path/to/file/component.test.js

Yesterday I got some new progress, by changing jest.config.testMatch to :

["./**/*.test.js"]

With this, stryker find tests to run, but if a put something like "./**/myComponent.test.js" no files are found, but file exists (If I could restrict test it would run faster to get mutators, cause we could modify some components and then pass their tests to mutate instead running all tests of project).

With this new configuration *.test.js It seems to try running test but crashes, saint about ECMA modules (some files uses import without beeing modules), then searching I found saying to use babel to transpile code,

Then created .babelrc

{
    "presets": [
        ["@babel/preset-env", { "modules": "auto" }]
    ],
    "plugins": [
        "@babel/plugin-syntax-jsx",
        [
            "@babel/plugin-proposal-private-property-in-object",
            {
                "loose": true
            }
        ]
    ]
}

add transform option to my jest.config.js

transform: {
    '^.+\\.js?$': 'babel-jest'
  },

Installed new packages:

"stryker-babel-transpiler": "^0.10.1", "stryker-javascript-mutator": "^0.14.1",

Apparently its is transpiling, but now when I run stryker run I got this error

Details:

/var/www/html/stryker-tmp/sandbox5175145/path/to/file/component.js:76
  return <_react2.ThemeProvider theme={theme}>{children}</_react2.ThemeProvider>;

SyntaxError: Unexpected token '<'
    at new Script (node:vm:100:7)
    at Runtime.createScriptFromCode (/var/www/html/node_modules/jest-runtime/build/index.js:1495:14)
    at Runtime._execModule (/var/www/html/node_modules/jest-runtime/build/index.js:1389:25)
    at Runtime._loadModule (/var/www/html/node_modules/jest-runtime/build/index.js:1013:12)
    at Runtime.requireModule (/var/www/html/node_modules/jest-runtime/build/index.js:873:12)
    at Runtime.requireModuleOrMock (/var/www/html/node_modules/jest-runtime/build/index.js:1039:21)
    at Object.require (/var/www/html/stryker-tmp/sandbox5175145/setup.react.js:4:1)
    at Runtime._execModule (/var/www/html/node_modules/jest-runtime/build/index.js:1429:24)
    at Runtime._loadModule (/var/www/html/node_modules/jest-runtime/build/index.js:1013:12)
    at Runtime.requireModule (/var/www/html/node_modules/jest-runtime/build/index.js:873:12)
    at jestAdapter (/var/www/html/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:65:15)
    at runTestInternal (/var/www/html/node_modules/jest-runner/build/runTest.js:281:16)
    at runTest (/var/www/html/node_modules/jest-runner/build/runTest.js:341:7)

Aparently this occurs when transpile the code, jest cannot run it, Am I in the rigth way? Or shoul we try to refactor files to be modules instead using babel? I got this in the end of day , so didnt advance so much, so I'll continue trying today

[AndreLZGava]

Hello lets do an update,

after a lot of tryes I got some progress but it till not working.

I used this option in my stryker config "testRunnerNodeArgs": ["--experimental-vm-modules"], didint work (I took babel of for the transpiler then).

Then I decided to go back into babel and now its like this:

{
    "presets": [
        "@babel/preset-react",
        "@babel/preset-typescript",
        [
            "@babel/preset-env",
            {
                "modules": "auto"
            }
        ]
    ],
    "plugins": [
        "@babel/plugin-syntax-jsx",
        [
            "@babel/plugin-proposal-private-property-in-object",
            {
                "loose": true
            }
        ]
    ]
}

I removed transform option from jest config, and now it runs but I have a problem saing that moduleNameMapper are not being found, (this part works great with jest but when running with stryker don't) This is my main problem for now. If I remove all moduleNameMaper my tests broke when runing stryke because my imports are not found.

Another part I reallized, at start of configuration I would like to run just one test for one mutation, that should be mutated and tested by stryker easilly and faster, but passing all the path to the components and its test just returns no test to be executed. Then I put this option into my test match, it takes more time and processing:

"jest": {
    "projectType": "custom",
    "enableFindRelatedTests": false,
    "configFile": "jest.config.js",
    "config": {
      "testMatch": ["./**/*.test.js"]
    }
  },

But doing with certain component or all path this found nothing to test (I verified and test has been copied into temp folder, but whe use ./**/MyComponent.test.js, no tests are executed, this probably would help me with time and mutation just what I want)

Now I'm stuck trying to run all tests and even with this I got error saying about import and export inside a node_module dir, this dir is configured in jest config transformIgnorePatterns: ['node_modules/?!(internPackage)'], and even with this I saw in log error that path reference to it isn't into tmp dir instead its full/project/path/into/node_modules/internPackage.

Any other tip with this problem with transcripting node_module or finding specific test from tmp dir?

[nicojs]

Installed new packages:

"stryker-babel-transpiler": "^0.10.1", "stryker-javascript-mutator": "^0.14.1",

Those packages are old and no longer used since... Stryker 3, back in 2020. Where did you find out about them?

Apparently its is transpiling, but now when I run stryker run I got this error

Details:

/var/www/html/stryker-tmp/sandbox5175145/path/to/file/component.js:76
  return <_react2.ThemeProvider theme={theme}>{children}</_react2.ThemeProvider>;

This is strange. Stryker should not transpile your code, just add mutant instrumentation. Do you think _react2.ThemeProvider was added by Stryker? We're using babel under the hood, so maybe it got transpiled without us noticing? This would be the first time we're seeing anything like this.

Then I decided to go back into babel and now its like this:

That is for the best. You really shouldn't be changing your testing approach with jest. It should simply work.

Maybe a final think you can try is to run Stryker with --inPlace. Maybe the simple fact of running Jest in the sub-directory causes the issue.

Is your project open source? If not, could you provide a small reproduction repository? Either as a .zip file or as a link to a github repo is fine.