Feature Request: support running in CIs

[elichai]

Hi, I think adding this crate to CI would be awesome, but it will require 3 changes:

1. Important - suppressing mutations, not all mutations can be fixed, and some are just plain fine, so being able to supress mutations either with a file (like sanitizers suppressions) or with attributes (like #[allow(...)]).

2. Important - most CI's fail when a command return a non 0 return code, so returning 0 when nothing was found and non-zero when something was found(survived/not covered).

3. Less important: Exiting on first "failure" (survived/not covered), usually CI's are built in a way to stop testing as soon as possible, so the best practice is to fail everything at the first error that you encounter.

Thank You!

[samuelpilz]

I'll go over the points

1. This has been on my List for a long time. Other mutation testing frameworks do this by selecting the list of mutations that should be active. This could be done by passing arguments to the #[mutate] attribute. I am not sure how this should be named. I thought of #[mutagen(exclude(mutator1, mutator2))] or #[mutagen(mutators=exclude(...))]. On the other side, #[mutagen(mutators=only(mutator1, mutator2))] could be used . I am not sure how this should be specified.

2. I don't quite understand. In which cases should which error code be used?

3. I am stronly against requring that all mutants have to be killed. However, we could add an option --require-mutation-score (e.g. cargo mutagen --require-mutation-score 90%) to that fails if the mutation score is below the given threshold.

@llogiq any thoughts?

[llogiq]

For code coverage, the score is usually posted somewhere to put on a badge or something, but I can see allowing for a threshold that needs to be reached to pass may be useful to some. I think the default threshold should be 0% and we should advise people to avoid chasing 100% coverage.

[samuelpilz]

The posting of the score is currently out-of-scope for this project. Someone would need to host that.

@llogiq I would be interested in your ideas on how users should specify a blacklist/whitelist of mutators inside the mutate attribute. I am totally unsure how to name the options