stryker-mutator
Provide fields for column/totals description
The schema has a free-form totals dictionary with header, value keypair. Since it is up to the report generator to decide the fields that should be available in the report, I would also like to be able to provide a description per column which could be displayed for example by hovering over the column header. We need this because we have some totals concepts such as caught mutants (killed by test vs timeout) vs killed mutants (killed by test failure).
Hmm interesting. But I think we should not create concepts per flavor of mutation testing framework. I think it would be helpful for people of concepts are shared. For example, everyone knows what branch coverage, line coverage, and function coverage is, no matter the code coverage analysis tool that you use.
We actually started this on our FAQ page. It is still online, but you cannot navigate there from our website (an oversight 🙈): https://stryker-mutator.io/faq
If you agree, I suggest creating a PR to add these concepts to the Stryker-handbook.
To improve the report, we can add links to the handbook page explaining the concepts automatically for concepts we agreed upon and we can also add a small description.
But if all concepts between frameworks are the same, then the totals columns should not be decided by the framework, they should be required.
Well... making the fields required might not make sense. "NoCoverage" for example is not supported by all. But maybe we should enforce the names. My idea was to support custom fields like "number of tests without mutation coverage" in the future while keeping the schema the same.
@stryker-mutator/stryker4s @stryker-mutator/contributors @stryker-mutator/stryker-net
What should we do here? Maybe we can have a talk about this later today?
@nicojs Yes but if you have custom fields then you need to be able to provide a description, as "number of tests without mutation coverage" is basically a description and is way too long as a column header :)
Agreed, once we have custom fields we should do that. However, the new flat-file syntax of the mutation testing report JSON no longer allows for custom columns. Or am I missing something here?
Ok, we should document the counters in the stryker-handbook and link from it in the report.
- killed
- timeout
- survived
- noCoverage
- runtimeErrors
- compileErrors
- totalDetected
- totalUndetected
- totalCovered
- totalValid
- totalInvalid
- totalMutants
- mutationScore
- mutationScoreBasedOnCoveredCode
Maybe we can leave a small description in the report as well.
It is documented here: https://github.com/stryker-mutator/stryker-handbook/blob/master/mutant-states-and-metrics.md#mutant-states-and-metrics
Now, all we need to do is to link to it from the mutation testing report.