jonathanmetzman

Hmm...I guess we could also use the blacklisting feature to disable incompatible fuzzers. and it would keep the API thinner. Let me think about this.

I like the idea of having benchmarks opt-in to be part of a "group" (e.g. rust) and then fuzzers can say which groups they want to be fuzz.

Can we start with blacklisting fuzzers for now? I'm happy to accept rust targets, though I'm not sure there's real benefit (hard to believe we could write a fuzzer better/worse...

Maybe start from the corpus for larger programs in future competitons.

So is the question that one of your benchmarks needs a shared object that we don't use during measuring? I'd recommend using static linking to get around this.

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-06-fishfuzz --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-06-fishfuzz-bug --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

Let me run this tomorrow.

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-08-04-fafuzz --fuzzers fafuzz afl aflplusplus_407 mopt --benchmarks bloaty_fuzz_target curl_curl_fuzzer_http freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer jsoncpp_jsoncpp_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer mbedtls_fuzz_dtlsclient openssl_x509 systemd_fuzz-link-parser vorbis_decode_fuzzer zlib_zlib_uncompress_fuzzer re2_fuzzer sqlite3_ossfuzz woff2_convert_woff2ttf_fuzzer

Looks like there's a bug with our experiments I got to fix first.