fuzzbench

fuzzbench copied to clipboard

Hi dongge,

@Alan32Liu I developed a variant of FishFuzz (USENIX Security23) to make it compatible with fuzzbench (the original version in the paper rely on LTO mode, which fails/timeout on loots of fuzzbench targets), and I would like to request an evaluation to see if it works. Could you help me run the fuzzer aflplusplus_ff_cmp, aflplusplus_fishfuzz and aflplusplus_fishfuzz_allbb?

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-04-ff --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_allbb

Thanks! Han

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-05-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_allbb

Hi @kdsjZh Thanks for writing down the command! That makes my work a lot easier : )

Just two minor notes:

1. The --experiment-name and --fuzzers parameters need to be swapped with your values (See the example command above)
2. We need to make a trivial modification to service/gcbrun_experiment.py to launch experiments in this PR. Here is an example to add a dummy comment : )

Please feel free to ping me once you finished 2. Thanks!

Hi dongge,

Thanks for the reminding. I've finished the dummy comment.

@Alan32Liu

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-05-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_allbb

Experiment 2023-10-05-fishfuzz data and results will be available later at: The experiment data. The experiment report.

Hello Dongge @Alan32Liu,

I fixed some build errors in libpcap/zlib and optimized the exploration stage, could you help me run the aflplusplus_ff_cmp, aflplusplus_fishfuzz_allbb and aflplusplus_fishfuzz_exp again?

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-12-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_exp aflplusplus_fishfuzz_allbb

Thanks!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-12-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_exp aflplusplus_fishfuzz_allbb

Experiment 2023-10-12-fishfuzz data and results will be available later at: The experiment data. The experiment report.

Hi Dongge @Alan32Liu ,

I fix the builder script/exploration stage and updated the llvm-12 to llvm-15.0.0 (to be coherent with fuzzbench's). Besides, existing fuzzers are working on ASan version binary, I want to include non-ASan to compare with existing main results. Could you help me start another campaign? Thank!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-19-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_allbb aflplusplus_fishfuzz_noasan aflplusplus_fishfuzz_noasan_all

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-21-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz_allbb aflplusplus_fishfuzz_noasan aflplusplus_fishfuzz_noasan_all

Experiment 2023-10-21-fishfuzz data and results will be available later at: The experiment data. The experiment report.

Hi Dongge @Alan32Liu ,

I'm plaining to do an ablation study about how each component works, with aflpp's tracepc option only (I found one possible bug with cmplog feature, so I opt for the tracepc only). Therefore I'm wondering could you help me with that? Many thanks for your patience and help!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-25-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-25-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

Sure! Experiment 2023-10-25-fishfuzz data and results will be available later at: The experiment data. The experiment report.

Hi @Alan32Liu Dongge,

I profile the fuzzer and find out that the sampling in exploitation stage has super high overhead, therefore I reduce the sampling frequency a bit and want another round of campaign if possible. Thank!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-11-01-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-11-01-fishfuzz --fuzzers aflplusplus_ff_cmp aflplusplus_fishfuzz aflplusplus_fishfuzz_noexploit

Experiment 2023-11-01-fishfuzz data and results will be available later at: The experiment data. The experiment report.

Hi Dongge,

@Alan32Liu Recently, I contacted Marc and located the cmplog issue. I integrated the fix and rewritten fishpp based on the latest AFL++. Could you help me with a dry run to test if my fix/latest integration is correct?

BTW, recently Marc tried the Fish++ vs AFL++ on bug benchmark, results shows that Fish++ is able to improve based on AFL++ baseline (ff_cmp_3). So I would like to request a bug evaluation as well.

# coverage
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-04-fishfuzz --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp
# bug
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-04-fishfuzz-bug --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

Thanks!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-06-fishfuzz --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-06-fishfuzz-bug --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

Experiment 2023-12-06-fishfuzz data and results will be available later at: The experiment data. The experiment report.

Experiment 2023-12-06-fishfuzz-bug data and results will be available later at: The experiment data. The experiment report.

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-06-fishfuzz --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-06-fishfuzz-bug --fuzzers fishpp_new fishpp_new_nocmp aflplusplus aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

Hi @Alan32Liu ,

I analyzed the data from this experiment and found that the path frequency sampling introduced notable overhead (in execution speed), while Fishpp already implemented a cheaper alternative in its exploitation mode. Therefore I added an explore mode Fishpp to avoid this overhead.

Could you help me run another campaign to verify? Thanks!

# cov (we only need cmplog version)

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-11-fishfuzz --fuzzers fishpp_new fishpp_new_exp aflplusplus 

# bug (nocmp seems to be better in some scenarios, therefore we need both cmp and nocmp for Fishpp explore)
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-11-fishfuzz-bug --fuzzers fishpp_new_exp aflplusplus fishpp_new_nocmp_exp aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-11-fishfuzz --fuzzers fishpp_new fishpp_new_exp aflplusplus

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-11-fishfuzz-bug --fuzzers fishpp_new_exp aflplusplus fishpp_new_nocmp_exp aflplusplus_nocmp --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

Experiment 2023-12-11-fishfuzz data and results will be available later at: The experiment data. The experiment report.

Experiment 2023-12-11-fishfuzz-bug data and results will be available later at: The experiment data. The experiment report.

Hi Dongge @Alan32Liu ,

I checked the data in existing reports and found out that h264 is not run.

The reports (both standard-cov and recent ff reports) are using cached data. e.g., in data.csv.gz from 2023-12-11-standard-cov, the data of openh264_decoder_fuzzer afl is copied from 2023-09-21-libafl. while aflfast's data are copied from 2023-03-20-ecofuzz.

BTW, I also notice that the aflplusplus is also not evaluated in 2023-12-11-standard-cov. and in the report, they cached results from 2023-12-10-aflpp, which uses a different aflplusplus commit. I don't know if it's an intended behavior. So this is just a head up.

I just updated the fishpp to the AFL++ 4.09c. could you help us do another round of test? Thanks!

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-15-fishfuzz --fuzzers fishpp_new fishpp_new_exp fishpp_new_fast aflplusplus


/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-15-fishfuzz-bug --fuzzers fishpp_new_exp aflplusplus fishpp_new_fast fishpp_new --benchmarks bloaty_fuzz_target_52948c harfbuzz_hb-shape-fuzzer_17863b libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb

The reports (both standard-cov and recent ff reports) are using cached data. e.g., in data.csv.gz from 2023-12-11-standard-cov, the data of openh264_decoder_fuzzer afl is copied from 2023-09-21-libafl. while aflfast's data are copied from 2023-03-20-ecofuzz.

BTW, I also notice that the aflplusplus is also not evaluated in 2023-12-11-standard-cov. and in the report, they cached results from 2023-12-10-aflpp, which uses a different aflplusplus commit. I don't know if it's an intended behavior. So this is just a head up.

Using data from old experiments is expected and default behavior. If a fuzzer-benchmark pair is not modified, there is no solid reason for re-generating its result, which can be expensive and time-consuming.

If you'd like to re-run all fuzzer-benchmark pairs, then you need to change this config. However, this is strongly discouraged due to the reasons above.

If you'd like to use a different version of a fuzzer, then please commit/merge the version into this PR. This protects your experiment from unexpected changes from other fuzzers, and ensures a clean and easy-to-access experiment environment/baseline for each report.