TechEmpower
Warn if frameworks exceed the maximum number of test mutations
According to https://github.com/TechEmpower/FrameworkBenchmarks/issues/8420 test mutations should be limited.
The number of test mutations will be limited to 10.
~This change makes mutations fail for frameworks if the maximum is reached.~ This change shows a warning message for frameworks if the maximum is reached.
I think what I would prefer is that users would get a warning when working on their implementation locally - specifically, it should call out that the max has already been reached, and if they are still set on getting their framework into the tests they need to contact maintainers and remove deprecated or discarded ones to make room.
This puts a lot of unfair effort on the part of a potential contributor to do more than just add their framework... I just don't have a better solution off hand.
@msmith-techempower I'm showing a warning now, similar to the warning if the "default" test variant is missing.
Why not run only 10 mutations as maximum per run.
Using the first 7-8 mutations always, and the rest will be run randomly in each run.
If any framework want a mutation than always run, they need to place in the first mutations in the benchmark_config.json.
Also we can add a new tag in the json, to define mutations than will run randomly selected. Showing the warning always, of course.
