Difference between "coverageanalysis" and "enableFindRelatedTests"?

[The-Punk-and-The-Monk]

Hello, I have been using stryker and jest-runner on my project for a while. A great tool for testing code. But as my code grows, I have to do some optimization on it. I noticed that there are two params related:

1. coverageAnalysis
2. enableFindRelatedTests

The interpretation of two params confusing me. Look like both of them can achieve the goal of only running the test case that covers the mutant. But, do these two params affect each other? How could I set the proper value to these two params?

1. coverageAnalysis = perTest; enableFindRelatedTests=true
2. coverageAnalysis = off; enableFindRelatedTests=true
3. coverageAnalysis = perTest; enableFindRelatedTests=false

[simondel]

Good question! In case of performance I would expect the following (fastest to slowest):

1. ~🚀 coverageAnalysis = perTest; enableFindRelatedTests=false~
2. ~🏃‍♀️ coverageAnalysis = perTest; enableFindRelatedTests=true~ (actually swapped, see next comment)
3. 🐌 coverageAnalysis = off; enableFindRelatedTests=true

Stryker's coverage analysis will cause it to skip running tests all together if no matching test is found. This code coverage is calculated once, meaning it will probably be faster than enableFindRelatedTests since that has to be done each time tests are executed. This would mean that if you only enable one settings, coverageAnalysis would be the way to go.

Since coverageAnalysis does most (if not all) work that enableFindRelatedTests does, it might make turning both settings on slower than only turning coverageAnalysis on.

What are your findings in regards to combining these three options?

[nicojs]

Great response @simondel . In general it is correct, but 1 and 2 should be swapped. And I've added 1:

1. 🚀 coverageAnalysis = perTest; enableFindRelatedTests=true
2. 🏃‍♀️ coverageAnalysis = off; enableFindRelatedTests=true

2. 🏃‍♀️ coverageAnalysis = perTest; enableFindRelatedTests=false
3. 🐌 coverageAnalysis = off; enableFindRelatedTests=tr

With --findRelatedTests, jest will do a reverse lookup by scanning import statements from the tests files. Test files that don't import the file-under-test (directly or indirectly) are not even loaded. If you run with --coverageAnalysis perTest but findRelatedTests false, then all files will still be loaded.

Imagine you have a medium size project with 100 files under test. For each file you have exactly 1 unit test file. Running with findRelatedTests will dramatically improve performance, since during mutation testing only 2 files are loaded at the same time. Loading files is not free, it's actually pretty expensive. So findRelatedTests is a must here.

However, findRelatedTests only works on a file basis. So all tests are run in the test files that cover the file-under-test.

Now, imagine you have a small project with 1 file under test and 1 unit test file, but the file contains 1000 tests that all take 1 second to run. This would make --coverageAnalysis perTest very effective, because that makes it so that all tests are skipped that don't cover the particular mutant. However, findRelatedTests doesn't matter here.

I hope that clarifies it more. We might want to add this to our docs somewhere.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.