fuzzbench

fuzzbench copied to clipboard

In Fuzzbench, coverage reports do not start at the origin. Instead the Y-axis begins at the coverage achieved for the first data point (currently after 15 minutes). The initial seed corpus itself already achieves some initial coverage. It is easy to see that an arbitrary initial corpus achieves arbitrary initial coverage. Fuzzer performance should thus be measured relative to this initial coverage of the initial seed corpus. Otherwise, early saturation achieved by the fuzzer might be indistinguishable from saturation achieved by the seed corpus.

Do you think we should do this on every experiment? Or would doing it once when adding a benchmark and putting it here suffice?

Good point! Another question is how to indicate this on the coverage growth plot. One option would be to start the Y axis at seed coverage instead of 0, the other would be to start from 0 but draw a horizontal line at the height of the seed coverage.

It's actually good practice to let the coverage plot start at the origin. This would also put the coverage improvements into perspective. I like the idea of adding a horizontal line to indicate the initial coverage. However, if even the smallest increments are important for Fuzzbench users, starting the Y at the initial coverage value might actually be fine.