fuzzbench icon indicating copy to clipboard operation
fuzzbench copied to clipboard

Feature proposal: FuzzBench aware fuzzers

Open andreafioraldi opened this issue 5 years ago • 1 comments

Would be cool to have an API that fuzzers can use to report stats to FuzzBench. Fuzzers should modify their code to enable more insightful evaluations (like https://hexgolems.com/2020/08/on-measuring-and-visualizing-fuzzer-performance/) like coverage per generated testcase or even execs/sec to evaluate performance.

So FuzzBench will produce two types of experiments, the generic (like now) that works with off-the-shelf fuzzers, and the FuzzBench aware evaluation with only the fuzzers that implements the FuzzBench API.

You can for instance set an env variable like FUZZBENCH_LIB=path/to that defines the path to the C library with such API and then we (fuzzers devs) can adjust the build systems to link such lib and enable the code that does the logging for fuzzbench using the preprocessor.

In AFL, for instance, we can insert snipperts like the following (from https://github.com/google/AFL/blob/master/afl-fuzz.c#L4650):

#ifdef FUZZBENCH_BUILD
#include "fuzzbench.h"
#endif

EXP_ST u8 common_fuzz_stuff(char** argv, u8* out_buf, u32 len) {

  u8 fault;

#ifdef FUZZBENCH_BUILD
  fuzzbench_log_testcase(out_buf, len);
#endif 

  ...

}

andreafioraldi avatar Oct 14 '20 17:10 andreafioraldi

Please also check current cl - https://github.com/google/fuzzbench/pull/648 and provide feedback.

inferno-chromium avatar Oct 14 '20 21:10 inferno-chromium