Feature proposal: FuzzBench aware fuzzers
Would be cool to have an API that fuzzers can use to report stats to FuzzBench. Fuzzers should modify their code to enable more insightful evaluations (like https://hexgolems.com/2020/08/on-measuring-and-visualizing-fuzzer-performance/) like coverage per generated testcase or even execs/sec to evaluate performance.
So FuzzBench will produce two types of experiments, the generic (like now) that works with off-the-shelf fuzzers, and the FuzzBench aware evaluation with only the fuzzers that implements the FuzzBench API.
You can for instance set an env variable like FUZZBENCH_LIB=path/to that defines the path to the C library with such API and then we (fuzzers devs) can adjust the build systems to link such lib and enable the code that does the logging for fuzzbench using the preprocessor.
In AFL, for instance, we can insert snipperts like the following (from https://github.com/google/AFL/blob/master/afl-fuzz.c#L4650):
#ifdef FUZZBENCH_BUILD
#include "fuzzbench.h"
#endif
EXP_ST u8 common_fuzz_stuff(char** argv, u8* out_buf, u32 len) {
u8 fault;
#ifdef FUZZBENCH_BUILD
fuzzbench_log_testcase(out_buf, len);
#endif
...
}
Please also check current cl - https://github.com/google/fuzzbench/pull/648 and provide feedback.