retry tests with current fb

[vanhauser-thc]

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-07-aflpp --fuzzers aflplusplus_early aflplusplus_last

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-07-aflpp2 --fuzzers aflplusplus_early aflplusplus_last

[vanhauser-thc]

@DonggeLiu I have big troubles getting the benchmarks working.

locally for me everything compiles fine, e.g. re2_fuzzer:

$ make test-run-aflplusplus_early-re2_fuzzer
docker build \
--tag gcr.io/fuzzbench/builders/benchmark/re2_fuzzer \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from gcr.io/fuzzbench/builders/benchmark/re2_fuzzer \
--file benchmarks/re2_fuzzer/Dockerfile \
benchmarks/re2_fuzzer
[+] Building 12.2s (12/12) FINISHED                              docker:default
...
[*] Fuzzing test case #1848 (1882 total, 0 crashes saved, state: started :-), mode=explore, perf_score=300, weight=inf, favorite=1, was_fuzzed=0, exec_us=0, hits=0, map=337, ascii=0, run_time=0:00:00:14)...
INFO:root:Doing final sync.

but when building here I get

Step #2 - "aflplusplus_early-re2_fuzzer-builder-intermediate": #7 ERROR: executor failed running [/bin/sh -c apt-get install -y lsb-release software-properties-common gnupg wget]: exit code: 100
Step #2 - "aflplusplus_early-re2_fuzzer-builder-intermediate": ------
Step #2 - "aflplusplus_early-re2_fuzzer-builder-intermediate":  > [3/6] RUN apt-get install -y lsb-release software-properties-common gnupg wget:
Step #2 - "aflplusplus_early-re2_fuzzer-builder-intermediate": ------
Step #2 - "aflplusplus_early-re2_fuzzer-builder-intermediate": executor failed running [/bin/sh -c apt-get install -y lsb-release software-properties-common gnupg wget]: exit code: 100
Finished Step #2 - "aflplusplus_early-re2_fuzzer-builder-intermediate"
ERROR
ERROR: build step 2 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1
------------------

and that makes no sense because for one fuzzer the same two targets always build on fuzzbench (ossfuzz, openh264), and all other fail. but the other fuzzer instance succeeds in all targets. and the only difference between the two in builder.Dockerfile is:

@@ -41,7 +41,7 @@
 ENV LLVM_CONFIG=llvm-config-18
 
 # Download afl++.
-RUN git clone -b early https://github.com/AFLplusplus/AFLplusplus /afl && \
+RUN git clone -b last https://github.com/AFLplusplus/AFLplusplus /afl && \
     cd /afl && \
     true

do you have any idea what is going wrong? btw. you can kill all afl++ fuzzing instances ...

[DonggeLiu]

ERROR: executor failed running [/bin/sh -c apt-get install -y lsb-release software-properties-common gnupg wget]: exit code: 100

Could flaky network issues cause this?

In the past, I recall seeing apt-get fail due to a network problem and then work again after a few hours in CI tests. Maybe re-try it in a few hours and see if it occurs again?

[vanhauser-thc]

I have this issue since Friday. And if it would be a network issue it would affect both fuzzers and random targets

[vanhauser-thc]

And you can see it works in the ci too - it’s green for most

[DonggeLiu]

I have this issue since Friday. And if it would be a network issue it would affect both fuzzers and random targets

I see. That's strange because I don't recall changing anything related last week. Unfortunately, I will need more time before I can debug this because I am currently occupied by other tasks.

For now, I can:

1. Cancel all AFL++ experiments, and
2. Re-launch the experiment here, just in case my account makes any difference (which is unexpected).

Meanwhile, I guess two potential ways may help us understand this error better:

1. Split the apt-get command and install one package in each. This helps us see which one caused the failure.
2. Add apt-get update && before the apt-get that caused the error. This should be unnecessary because your first RUN command already did it, but it can at least rule out a possibility.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-08-aflpp --fuzzers aflplusplus_early aflplusplus_last

[vanhauser-thc]

Same in your run. Ci is green for the targets, but for the fuzzing the same fuzzer only one target built successfully :( And only half of the targets are there. Weird.

the test is important because this is testing a major change for llvm 16+ and we need a release very soon

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-08-aflpp3 --fuzzers aflplusplus_early aflplusplus_last

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-09-aflpp --fuzzers aflplusplus_early aflplusplus_last

[vanhauser-thc]

@DonggeLiu it worked when I switched to llvm 16 (or the issue just dissolved for other reasons). trying llvm 19 now.

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-09-aflpp2 --fuzzers aflplusplus_early aflplusplus_last

[jonathanmetzman]

I'm guessing the differences between this happening in prod vs local are because of caching. I agree with Dongge that the issue looks like it is caused by not having apt-get update && before apt-get install

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-18-aflpp --fuzzers aflpp aflpp2

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-19-aflpp --fuzzers aflpp aflpp2

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-20-aflpp --fuzzers aflpp aflpp2

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-23-aflpp --fuzzers aflplusplus aflplusplus_weight0 aflplusplus_weight1

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-24-aflpp --fuzzers aflplusplus aflplusplus_weight0 aflplusplus_aweight0 aflplusplus_aweight1 --benchmarks bloaty_fuzz_target curl_curl_fuzzer_http freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer jsoncpp_jsoncpp_fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpcap_fuzz_both libpng_libpng_read_fuzzer libxml2_xml libxslt_xpath mbedtls_fuzz_dtlsclient openh264_decoder_fuzzer openssl_x509 openthread_ot-ip6-send-fuzzer proj4_proj_crs_to_crs_fuzzer re2_fuzzer sqlite3_ossfuzz stb_stbi_read_fuzzer systemd_fuzz-link-parser vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

[vanhauser-thc]

@jonathanmetzman this is what I meant with the issues I have on fuzzbench:

Everything built fine for https://www.fuzzbench.com/reports/experimental/2024-05-23-aflpp/index.html

in https://www.fuzzbench.com/reports/experimental/2024-05-24-aflpp/index.html I didn’t change these but added two more. The ones I added are fine but the original two now have one target that didn’t built.

I didn’t check the build logs what exactly went wrong but either way it is something that fuzzbench should detect and retry, wiping a cache beforehand etc or whatever is causing this.

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-27-aflpp --fuzzers aflplusplus_data0 aflplusplus_data1 aflplusplus_data2 --benchmarks bloaty_fuzz_target curl_curl_fuzzer_http freetype2_ftfuzzer harfbuzz_hb-shape-fuzzer jsoncpp_jsoncpp_fuzzer lcms_cms_transform_fuzzer libjpeg-turbo_libjpeg_turbo_fuzzer libpcap_fuzz_both libpng_libpng_read_fuzzer libxml2_xml libxslt_xpath mbedtls_fuzz_dtlsclient openh264_decoder_fuzzer openssl_x509 openthread_ot-ip6-send-fuzzer proj4_proj_crs_to_crs_fuzzer re2_fuzzer sqlite3_ossfuzz stb_stbi_read_fuzzer systemd_fuzz-link-parser vorbis_decode_fuzzer woff2_convert_woff2ttf_fuzzer zlib_zlib_uncompress_fuzzer

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-28-aflpp --fuzzers aflplusplus aflplusplus_420c aflplusplus_410c

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-28-aflpp --fuzzers aflplusplus_old aflplusplus_420c aflplusplus_410c aflplusplus_420a --benchmarks bloaty_fuzz_target curl_curl_fuzzer_http lcms_cms_transform_fuzzer libpcap_fuzz_both mbedtls_fuzz_dtlsclient openthread_ot-ip6-send-fuzzer proj4_proj_crs_to_crs_fuzzer systemd_fuzz-link-parser

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-30-aflpp --fuzzers aflplusplus_old aflplusplus_420c aflplusplus_410c aflplusplus_420a --benchmarks bloaty_fuzz_target curl_curl_fuzzer_http lcms_cms_transform_fuzzer libpcap_fuzz_both mbedtls_fuzz_dtlsclient openthread_ot-ip6-send-fuzzer proj4_proj_crs_to_crs_fuzzer systemd_fuzz-link-parser

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-30-aflpp2 --fuzzers aflplusplus_420c aflpp_0 aflpp_1 aflpp_2 aflpp_3 aflpp_4 --benchmarks bloaty_fuzz_target curl_curl_fuzzer_http lcms_cms_transform_fuzzer libpcap_fuzz_both mbedtls_fuzz_dtlsclient openthread_ot-ip6-send-fuzzer proj4_proj_crs_to_crs_fuzzer systemd_fuzz-link-parser

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-31-aflpp2 --fuzzers aflplusplus_420c aflpp_0 aflpp_1 aflpp_2 aflpp_3 --benchmarks bloaty_fuzz_target curl_curl_fuzzer_http lcms_cms_transform_fuzzer libpcap_fuzz_both mbedtls_fuzz_dtlsclient openthread_ot-ip6-send-fuzzer proj4_proj_crs_to_crs_fuzzer systemd_fuzz-link-parser

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-31-aflpp --fuzzers aflplusplus_420c aflpp_0 aflpp_1 aflpp_2 aflpp_3 aflpp_4 aflpp_5 aflpp_6 aflpp_7 aflpp_8 aflpp_9 aflpp_10 aflpp_11 --benchmarks bloaty_fuzz_target libpcap_fuzz_both proj4_proj_crs_to_crs_fuzzer

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-01-aflpp --fuzzers aflpp_0 aflpp_1 aflpp_10 aflpp_11 --benchmarks bloaty_fuzz_target libpcap_fuzz_both proj4_proj_crs_to_crs_fuzzer

[vanhauser-thc]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-06-01-aflpp2 --fuzzers aflplusplus aflplusplus_reg --benchmarks bloaty_fuzz_target libpcap_fuzz_both proj4_proj_crs_to_crs_fuzzer

[vanhauser-thc]

@DonggeLiu you can cancel all running benchmarks by me, I found the regression and know what to fix. btw if I would have the ability to cancel my benchmarks on my own that would be great, it would save resources because sometimes I do not need full runs or fuck one up and the important fuzzer does not run etc.