google
AFLChurn++ Experiment targeting bug-based benchmarks
A follow on from https://github.com/google/fuzzbench/pull/2012
Testing AFLChurn++ on this old branch with bug-based benchmarks
@DonggeLiu you mentioned that there are 15 bug-based benchmarks. As there are more than 15 benchmarks, do you know which ones they are?
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-12-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus
@DonggeLiu you mentioned that there are 15 bug-based benchmarks. As there are more than 15 benchmarks, do you know which ones they are?
It's on this tag: https://github.com/google/fuzzbench/tree/SBFT'23.
Bug-based benchmark directories end with the commit id, and its yaml file has type: bug
gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-12-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus
Do you want to run this on the bug-based or coverage based?
gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-12-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus
Do you want to run this on the bug-based or coverage based?
On the bug based, thanks!
On the bug based, thanks!
Then would you please rebase this PR on the following tag, which has the bug-based benchmarks? The current base branch (master) does not have those benchmarks. Thanks.
It's on this tag: https://github.com/google/fuzzbench/tree/SBFT'23. Bug-based benchmark directories end with the commit id, and its yaml file has
type: bug
Then would you please rebase this PR on the following tag
I did rebase my fork of fuzzbench on that tag, but I'm unsure how to target a tag in a PR, I'm only able to target branches (not tags)
Is there something else I should be doing?
Hmm, I will need an extra config to allow PR experiments when the base is not master
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-13-aflchurnplusplus --fuzzers aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_1a34a0
Experiment 2024-08-13-aflchurnplusplus data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
Once the test experiment is done (and if all looks well) can you please run the following:
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_1a34a0 aspell_aspell_fuzzer_e8eb74 assimp_assimp_fuzzer_4d451f bloaty_fuzz_target_52948c ffmpeg_ffmpeg_demuxer_fuzzer_7adeef file_magic_fuzzer_2d5f85 grok_grk_decompress_fuzzer_9cd001 harfbuzz_hb-shape-fuzzer_17863b lcms_cms_transform_all_fuzzer_97d37d libaom_av1_dec_fuzzer_6e1848 libpcap_fuzz_filter_98b0a2 libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb systemd_fuzz-network-parser_288baf
That includes all the bug benchmarks. I would like to run the test on AFL and AFL++ as well, so I have data to make direct comparisons
The report looks good so far, I will start the new one now to minimize our waiting time.
But since you have modified afl++, could you please set this to false so that it won't merge the old result into your report?
https://github.com/google/fuzzbench/blob/d8a77233f3919477648c98d2c8228b530b7036ba/service/experiment-config.yaml#L18
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_1a34a0 aspell_aspell_fuzzer_e8eb74 assimp_assimp_fuzzer_4d451f bloaty_fuzz_target_52948c ffmpeg_ffmpeg_demuxer_fuzzer_7adeef file_magic_fuzzer_2d5f85 grok_grk_decompress_fuzzer_9cd001 harfbuzz_hb-shape-fuzzer_17863b lcms_cms_transform_all_fuzzer_97d37d libaom_av1_dec_fuzzer_6e1848 libpcap_fuzz_filter_98b0a2 libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb systemd_fuzz-network-parser_288baf
Experiment 2024-08-14-aflchurnplusplus data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
As AFLChurn++ is a regression fuzzer, I would also like to be able to compare with results of fuzzing on the bug-introducing commit. I have added new benchmarks for benchmarks previously tested but changed the commit they are using. If the CIs pass, can you please run this for me:
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_eee13b aspell_aspell_fuzzer_aed7cd assimp_assimp_fuzzer_bdee65 bloaty_fuzz_target_f01ea5 ffmpeg_ffmpeg_demuxer_fuzzer_fe85af grok_grk_decompress_fuzzer_d9ff920 lcms_cms_transform_all_fuzzer_a9796f libpcap_fuzz_filter_bc594f
Thank you!
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_eee13b aspell_aspell_fuzzer_aed7cd assimp_assimp_fuzzer_bdee65 bloaty_fuzz_target_f01ea5 ffmpeg_ffmpeg_demuxer_fuzzer_fe85af grok_grk_decompress_fuzzer_d9ff920 lcms_cms_transform_all_fuzzer_a9796f libpcap_fuzz_filter_bc594f
Experiment 2024-08-17-aflchurnplusplus data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
@DonggeLiu the in progress experiment report doesn't appear to be working, is there a problem with the experiment?
@DonggeLiu the in progress experiment report doesn't appear to be working, is there a problem with the experiment?
Yes, there appears to be a quota error, based on the base-image build log. I reckon this is flaky, let me re-try.
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-18-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_eee13b aspell_aspell_fuzzer_aed7cd assimp_assimp_fuzzer_bdee65 bloaty_fuzz_target_f01ea5 ffmpeg_ffmpeg_demuxer_fuzzer_fe85af grok_grk_decompress_fuzzer_d9ff920 lcms_cms_transform_all_fuzzer_a9796f libpcap_fuzz_filter_bc594f
Experiment 2024-08-18-aflchurnplusplus data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
Hey, just a couple changes to benchmarks so that they're valid for my analysis. Can you please run this for me (afl has been intentionally excluded):
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-31-aflchurnplusplus --fuzzers aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_1a34a0 arrow_arrow-ipc-stream-fuzz_eee13b libaom_av1_dec_fuzzer_5a4757 libaom_av1_dec_fuzzer_cbbc5c systemd_fuzz-network-parser_46c3b1 systemd_fuzz-network-parser_82d7a1
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-31-aflchurnplusplus --fuzzers aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_1a34a0 arrow_arrow-ipc-stream-fuzz_eee13b libaom_av1_dec_fuzzer_5a4757 libaom_av1_dec_fuzzer_cbbc5c systemd_fuzz-network-parser_46c3b1 systemd_fuzz-network-parser_82d7a1