Another LibAFL compaison on new programs of SBFT'23

[tokatoka]

#1890 Created another PR for this

@Alan32Liu I added build files for the additional 17 fuzzers in this branch.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt --experiment-name 2023-09-02-libafl-1

[DonggeLiu]

OK, I will launch the experiments as we discussed in #1890. I hope you are feeling better now : )

[DonggeLiu]

Experiment 2023-09-02-libafl-1 data and results will be available later at: The experiment data. The experiment report.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_naive libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram4 libafl_fuzzbench_ngram8 --experiment-name 2023-09-02-libafl-2

[DonggeLiu]

Experiment 2023-09-02-libafl-2 data and results will be available later at: The experiment data. The experiment report.

[tokatoka]

seems like one of the rust dependency is complaining an error in CI...

let me fix this first

[tokatoka]

now it looks good

could you start the experiment again?

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt --experiment-name 2023-09-02-libafl-1

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt --experiment-name 2023-09-03-libafl-1

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_cov_accounting libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt --experiment-name 2023-09-04-libafl-1

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_naive libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram4 libafl_fuzzbench_ngram8 --experiment-name 2023-09-04-libafl-2

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_rand_scheduler libafl_fuzzbench_value_profile libafl_fuzzbench_weighted--experiment-name 2023-09-04-libafl-3

[DonggeLiu]

Experiment 2023-09-04-libafl-1 data and results will be available later at: The experiment data. The experiment report.

[DonggeLiu]

Experiment 2023-09-04-libafl-2 data and results will be available later at: The experiment data. The experiment report.

[DonggeLiu]

Experiment 2023-09-04-libafl-3 data and results will be available later at: The experiment data. The experiment report.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_rand_scheduler libafl_fuzzbench_value_profile libafl_fuzzbench_weighted --experiment-name 2023-09-05-libafl-3

[DonggeLiu]

Sorry, there is a type in my command to run 2023-09-04-libafl-3. It's fixed above in experiment 2023-09-05-libafl-3, its data and results will be available later at: The experiment data. The experiment report.

[tokatoka]

Hi @Alan32Liu

I see there's some missing data, NaNs in the experiment result. https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2023-09-04-libafl-1/index.html https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2023-09-04-libafl-2/index.html These two have 9 and 3 NaNs respectively. I don't think this is a bug in the fuzzer because it ran perfectly in the usual 23 benchmarks before

I guess maybe 5 fuzzers are still too many and they couldn't generate coverage data in the end. I want to get the data missing in NaNs. For this purpose, Could you do the following experiment?

Run1 (For missing values in libafl_fuzzbench_cmplog and libafl_fuzzbench_explore)

- assimp_assimp_fuzzer
- brotli_decode_fuzzer
- guetzli_guetzli_fuzzer
- libcoap_pdu_parse_fuzzer

for

- libafl_fuzzbench_cmplog
- libafl_fuzzbench_explore

Run2 (For missing values in libafl_fuzzbench_fast, libafl_fuzzbench_cov_accounting, and libafl_fuzzbench_mopt)

- astc-encoder_fuzz_astc_physical_to_symbolic
- double-conversion_string_to_double_fuzzer
- draco_draco_pc_decoder_fuzzer
- fmt_chrono-duration-fuzzer
- libcoap_pdu_parse_fuzzer
- libhevc_hevc_dec_fuzzer

for

- libafl_fuzzbench_fast
- libafl_fuzzbench_cov_accounting
- libafl_fuzzbench_mopt

Run3 (For missing values in libafl_fuzzbench_ngram8 and libafl_fuzzbench_naive_ctx)

- brotli_decode_fuzzer
- libaom_av1_dec_fuzzer

for

- libafl_fuzzbench_ngram8
- libafl_fuzzbench_naive_ctx

[tokatoka]

The commands would be

Run1: /gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks assimp_assimp_fuzzer brotli_decode_fuzzer guetzli_guetzli_fuzzer libcoap_pdu_parse_fuzzer --fuzzers libafl_fuzzbench_cmplog libafl_fuzzbench_explore --experiment-name 2023-09-06-libafl-1

Run2: /gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks astc-encoder_fuzz_astc_physical_to_symbolic double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer fmt_chrono-duration-fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer --fuzzers libafl_fuzzbench_cov_accounting libafl_fuzzbench_fast libafl_fuzzbench_mopt --experiment-name 2023-09-06-libafl-2

Run3: /gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks brotli_decode_fuzzer libaom_av1_dec_fuzzer --fuzzers libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram8 --experiment-name 2023-09-06-libafl-3

[tokatoka]

oh wait. we do actually have a perfect experiment here: https://www.fuzzbench.com/reports/experimental/2023-09-03-libafl-1/index.html for

- libafl_fuzzbench_cmplog
- libafl_fuzzbench_explore
- libafl_fuzzbench_fast
- libafl_fuzzbench_cov_accounting
- libafl_fuzzbench_mopt

Sorry I didn't notice this.

In this case could you only run Run3:

- brotli_decode_fuzzer
- libaom_av1_dec_fuzzer

for

- libafl_fuzzbench_ngram8
- libafl_fuzzbench_naive_ctx

?

[DonggeLiu]

Hi @Alan32Liu

I see there's some missing data, NaNs in the experiment result.

Hi @tokatoka, thanks for the summary. Unfortunately, some of them are due to build failures. For example benchmark-botan_tls_server-fuzzer-libafl_fuzzbench_cmplog. They are recorded in the build-logs (e.g., https://storage.googleapis.com/fuzzbench-data/index.html?prefix=<experiment_name>/build-logs/).

In this case could you only run Run3:

• brotli_decode_fuzzer
• libaom_av1_dec_fuzzer for
• libafl_fuzzbench_ngram8
• libafl_fuzzbench_naive_ctx

Sure, they build succifully, but got the following error during runitme:

Executed command: "nice -n 5 python3 -u -c from fuzzers.libafl_fuzzbench_ngram8 import fuzzer; fuzzer.fuzz("/out/seeds", "/out/corpus", "/out/av1_dec_fuzzer")" returned: 1.

I will rerun them below to double-check this is not flaky.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks brotli_decode_fuzzer libaom_av1_dec_fuzzer --fuzzers libafl_fuzzbench_naive_ctx libafl_fuzzbench_ngram8 --experiment-name 2023-09-06-libafl-3

[tokatoka]

Unfortunately, some of them are due to build failures.

Yes for botan-tls, the issue was that their build system doesn't recognize our compiler wrapper.

Sure, they build succifully, but got the following error during runitme:

Yes this seems to be a problem in our ngram instrumentation.

I will rerun them below to double-check this is not flaky.

Thanks, https://www.fuzzbench.com/reports/experimental/2023-09-06-libafl-3/index.html this result is good for me.

[DonggeLiu]

Thanks, https://www.fuzzbench.com/reports/experimental/2023-09-06-libafl-3/index.html this result is good for me.

OK, I will post the data link below for future reference.

Experiment 2023-09-06-libafl-3 data and results will be available later at: The experiment data. The experiment report.

[tokatoka]

Hi @Alan32Liu

Actually last time I forgot to add one fuzzer; libafl_fuzzbench_grimoire.. :disappointed: Could you run the experiment for this fuzzer on all of the 17 target please?

FIY, The command would be

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_grimoire --experiment-name 2023-09-18-libafl

Thanks.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --benchmarks arduinojson_json_fuzzer assimp_assimp_fuzzer astc-encoder_fuzz_astc_physical_to_symbolic botan_tls_server brotli_decode_fuzzer double-conversion_string_to_double_fuzzer draco_draco_pc_decoder_fuzzer dropbear_fuzzer-postauth_nomaths firestore_firestore_serializer_fuzzer fmt_chrono-duration-fuzzer guetzli_guetzli_fuzzer icu_unicode_string_codepage_create_fuzzer jansson_json_load_dump_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer libhevc_hevc_dec_fuzzer librdkafka_fuzz_regex --fuzzers libafl_fuzzbench_grimoire --experiment-name 2023-09-18-libafl

[DonggeLiu]

Hi @Alan32Liu

Actually last time I forgot to add one fuzzer; libafl_fuzzbench_grimoire.. 😞 Could you run the experiment for this fuzzer on all of the 17 target please?

No problem : ) Experiment 2023-09-18-libafl data and results will be available later at: The experiment data. The experiment report.

[tokatoka]

Hi @Alan32Liu

I guess this is the last experiment request. :flushed: I want to run two fuzzers.

1. We found a bug in our libafl_fuzzbench_grimoire fuzzer. Now I fixed this, I want to run this fuzzer on the usual 23 benchmarks.

The command would be

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-libafl-grimoire --fuzzers libafl_fuzzbench_grimoire

2. We have made another fuzzer libafl_fuzzbench_composition, I would like to run these on these 6 programs.

- assimp_assimp_fuzzer
- brotli_decode_fuzzer
- draco_draco_pc_decoder_fuzzer
- guetzli_guetzli_fuzzer
- libaom_av1_dec_fuzzer
- libcoap_pdu_parse_fuzzer

The command would be

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-libafl-grimoire --benchmarks assimp_assimp_fuzzer brotli_decode_fuzzer draco_draco_pc_decoder_fuzzer guetzli_guetzli_fuzzer libaom_av1_dec_fuzzer libcoap_pdu_parse_fuzzer --fuzzers libafl_fuzzbench_composition --experiment-name 2023-09-25-libafl-composition

Thanks

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-libafl-grimoire --fuzzers libafl_fuzzbench_grimoire

[DonggeLiu]

Experiment 2023-09-25-libafl-grimoire data and results will be available later at: The experiment data. The experiment report.