bolero
bolero copied to clipboard
Stats: marking known-useless runs
Assume a fuzzer that looks like:
#[test]
fn fuzzer() {
check!().for_each(|i: &[u8]| {
if let Ok(i) = parse_input(i) {
the_thing_we_actually_want_to_fuzz(i)
}
})
}
Basically, any run that does not result in parse_input
returning Ok
is useless, as the goal of this fuzzer is not semantically to fuzz parse_input
(which may be ad-hoc for this fuzzer).
In turn, I'm thinking it might make sense to add a way to report to cargo-bolero that the input was actually (un)interesting, so that it could report stats (eg. "60% of all runs were uninteresting, consider improving parse_int to increase the proportion of valid inputs").
Or even, having the API ending up as something like:
#[test]
fn fuzzer() {
check!(min_interesting_input_rate=0.6).for_each(|interesting: &mut BoleroMetadata, i: &[u8]| {
if let Ok(i) = parse_input(i) {
interesting.is_interesting();
the_thing_we_actually_want_to_fuzz(i)
}
})
}
And then bolero when run in cargo-test
mode would run for 1000 times and assert that at least 600 of the runs hit the interesting.is_interesting()
line.
This would be bells and whistles, so definitely not essential, but WDYT about it?