stryker-mutator
a way to merge reports from multiple node versions?
Is your feature request related to a problem? Please describe. Specifically, I have lots of packages that do feature-detection - something that, eg, works in node 20 and not in node 18 - and so running the tests in node 20 won't "survive" mutation of that feature-detection precisely because it would require the older engine to test.
If I run the tests in node 18, then a different complementary set of code will "survive" mutation.
Describe the solution you'd like This problem occurs in code coverage as well, and the industry-wide solution there is to generate coverage reports as files on disk, and then to merge those after CI is done, and then report on the final result - meaning, in order for a mutation to have "survived", it has to have done so in every input to the merged final report.
Describe alternatives you've considered Asking for the ability to have "ignore mutation" comments or similar; manually ignoring results I know aren't a problem; or not using this mutation approach at all.
Additional context
[Survived] ConditionalExpression
index.js:1:14
- var hasMap = typeof Map === 'function' && Map.prototype;
+ var hasMap = true;
in an engine without native Map, this mutation will fail a great many tests; but in an engine with it, it will survive. This should not be reported, as long as I'm able to run the mutation on a node version without Map (or, as long as I'm able to run a "preamble" that deletes Map before running the tests and mutations).
Thank you for bringing this to our attention! We currently don't support this feature, but we'd certainly like to include this in our feature-set.
We should implement this similar to aggregateResultsByModule, I've therefore transferred the issue to this repository.
We'd certainly welcome your PR if you're interested in working on this yourself!
I don't think this necessitates a new feature in mutation testing elements, the suggestion you give to merge the reports afterwards can be implemented in your CI by using the existing metrics package to parse the json files into report objects, you can then merge the reports according to your specific merge rules and persist the report back to disk.
What specific merge rules? They'd be the same for everyone - a mutation can only "survive" if every single input had it survive.
You are probably right, I only generalized because there is always that one specific use case where a user does have an alternative perspective and wants the merge to happen in a different way.
My main point is that a merge should already be trivial using the metrics package without introducing a new concept similar to Module.
How can I do that, then? meaning, what commands do i run to generate a report with a zero exit code, and then later, what command do i run to take all the reports as input and evaluate the merged report with the resulting exit code?
Apologies if this was unclear, my comment was directed towards Andreas not you. Your feature request is imo a potential feature to consider in strykerJS and not mutation testing elements (where this issue was moved to). Alternatively this is a feature that could potentially be added to an existing report-merging tool. As a workaround in the mean time you could simply run stryker without threshold break so it generates the report with a zero exit code twice, then run a small script in your pipeline that merges the reports and throws an error code if the combined score is lower than your thresholds.
Gotcha, thanks :-) I have no opinion about where the logic lives, but it would be ideal for me to have it be in one or two public npm packages so I can npx them into CI.