bolero icon indicating copy to clipboard operation
bolero copied to clipboard

Stats: marking known-useless runs

Open Ekleog opened this issue 1 year ago • 2 comments

Assume a fuzzer that looks like:

#[test]
fn fuzzer() {
    check!().for_each(|i: &[u8]| {
        if let Ok(i) = parse_input(i) {
            the_thing_we_actually_want_to_fuzz(i)
        }
    })
}

Basically, any run that does not result in parse_input returning Ok is useless, as the goal of this fuzzer is not semantically to fuzz parse_input (which may be ad-hoc for this fuzzer).

In turn, I'm thinking it might make sense to add a way to report to cargo-bolero that the input was actually (un)interesting, so that it could report stats (eg. "60% of all runs were uninteresting, consider improving parse_int to increase the proportion of valid inputs").

Or even, having the API ending up as something like:

#[test]
fn fuzzer() {
    check!(min_interesting_input_rate=0.6).for_each(|interesting: &mut BoleroMetadata, i: &[u8]| {
        if let Ok(i) = parse_input(i) {
            interesting.is_interesting();
            the_thing_we_actually_want_to_fuzz(i)
        }
    })
}

And then bolero when run in cargo-test mode would run for 1000 times and assert that at least 600 of the runs hit the interesting.is_interesting() line.

This would be bells and whistles, so definitely not essential, but WDYT about it?

Ekleog avatar Aug 18 '22 19:08 Ekleog