cargo-mutants icon indicating copy to clipboard operation
cargo-mutants copied to clipboard

Notice test failures that accompany hangs

Open kpreid opened this issue 1 year ago • 1 comments

Consider a case where a mutation causes a hang, but is also detected by a test:

pub struct Iter(u32);
impl Iterator for Iter {
    type Item = u32;
    fn next(&mut self) -> Option<u32> {
        if self.0 == 0 {
            None
        } else {
            self.0 -= 1;
            Some(self.0)
        }
    }
}

#[test]
fn t1() {
    assert_eq!(Iter(10).next(), Some(9));
}

#[test]
fn t2() {
    for _ in Iter(10) {}
}

In this case, mutations of Iter::next() may cause t1 to fail and t2 to hang. This is counted as a timeout, but could be detected rapidly as a caught mutant, and this might make cargo-mutants more useful on codebases that have a lot of this kind of code (stopping conditions that can be trivially mutated into running forever).

Detecting this case would require parsing the test harness output which could sometimes be corrupted (stray logging to real stderr) or arbitrary (custom test harnesses), so it should be optional.

(In the example case above, it would arguably be wise for t2 to have its own maximum-iterations guard condition, but in the general case, the loop may be part of code that is not test code and cannot predict the maximum number of iterations.)

kpreid avatar Oct 17 '24 19:10 kpreid

Yep, https://github.com/sourcefrog/cargo-mutants/issues/194 has some other related ideas. I think the heart of it is: the standard Rust test framework does not exit early when one test fails, but at least for the cargo mutants use case it would be better if it did.

sourcefrog avatar Oct 17 '24 20:10 sourcefrog