google
DGFuzz Testing
This is a new fuzzer that I'd like to test out.
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-07-dgfuzz --fuzzers dgfuzz libafl
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-08-dgfuzz --fuzzers dgfuzz
Ops: Would you mind making a trivial modification to service/gcbrun_experiment.py? This will allow PR experiments. Here is an example to add a dummy comment : ) Thanks!
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-08-dgfuzz --fuzzers dgfuzz
@DonggeLiu Agh, forgot to run make presubmit and the linter had taken issue with my whitespace. I think it needs restarting? The experiment report didn't populate, so I assume it's down to the CI checks failing.
@DonggeLiu Agh, forgot to run
make presubmitand the linter had taken issue with my whitespace. I think it needs restarting?
Hi @DanBlackwell That experiment has launched: The data directory is not empty. The report normally takes longer to show after FB finishes building the benchmarks and fuzzers.
I've manually terminated all VM instances of that experiment and will relaunch a new one to use the latest code.
The experiment report didn't populate, so I assume it's down to the CI checks failing.
The CI checks won't block PR experiments, they are separated. But it's good that you double-checked CIs, as sometimes the fuzzer might not be compatible with some benchmarks : )
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-08-dgfuzz-1 --fuzzers dgfuzz
Hi @DanBlackwell That experiment has launched: The data directory is not empty. The report normally takes longer to show after FB finishes building the benchmarks and fuzzers.
My apologies, I thought the experiment-folders and coverage dirs in the data directory normally showed up after 1-2 hours; then when I saw the CI checks failed I jumped to the conclusion that it only ran the build and bailed.
But it's good that you double-checked CIs, as sometimes the fuzzer might not be compatible with some benchmarks : )
This fuzzer will only work on a subset for now, as DFSan requires all dependencies to be compiled with the -fsanitize=dataflow flag set (including the C++ standard library, OpenSSL etc.). If it looks promising then I'll put in some time to figure out how that can be done.
@DonggeLiu Hmm, seems like it’s not started the experiment again (no experiment-folders created again). Is it possible that because it’s the only fuzzer in the experiment - and some of the targets did not build?
If so, maybe it could be run with just the 8 targets that do build. I've now added dgfuzz to all the unsupported_fuzzers fields for those that do not build, so in theory it can be run with the same gcb command. (from what I saw here, it looks like gcbrun only lets you do one benchmark at a time, is that correct?)
Additionally, I’ve set the merge_with_nonprivate flag to false just in case running less programs causes any weirdness; it would be best if we could also run libafl as a baseline / control given that it won't merge now.
If it helps, these are the 8 that build ok:
harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libpcap_fuzz_both mbedtls_fuzz_dtlsclient openthread_ot-ip6-send-fuzzer stb_stbi_read_fuzzer vorbis_decode_fuzzer zlib_zlib_uncompress_fuzzer
PS, if the experiment did start at some point (it's been 5 hours without starting at the time of writing), then ignore all this and let it run through :)
PS, if the experiment did start at some point (it's been 5 hours without starting at the time of writing), then ignore all this and let it run through :)
I think it did : ) https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2024-07-08-dgfuzz-1/index.html
Is it possible that because it’s the only fuzzer in the experiment - and some of the targets did not build?
I don't think this will block the experiment. If this happens, then that benchmark will be excluded from the exp without affecting others.
Hi @DonggeLiu , I've pushed up some updates - I think that last run was getting killed due to shared memory not getting freed correctly. Can you run the following:
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-10-dgfuzz --fuzzers dgfuzz
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-11-dgfuzz --fuzzers dgfuzz
Experiment 2024-07-11-dgfuzz data and results will be available later at:
The experiment data.
The experiment report(experimental).
@DonggeLiu Just a note that most of these did not build with some apt errors it seems. The following was taken from libpcap_fuzz_both:
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.257 Get:240 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-dev amd64 3.8.2-0ubuntu2 [1212 B]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.447 Err:233 http://security.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04.9
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.447 404 Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.555 Err:234 http://security.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-dev amd64 3.8.10-0ubuntu1~20.04.9
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.555 404 Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.555 Err:239 http://security.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-dev amd64 3.8.10-0ubuntu1~20.04.9
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.555 404 Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.562 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/python3.8/libpython3.8_3.8.10-0ubuntu1~20.04.9_amd64.deb 404 Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.562 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/python3.8/libpython3.8-dev_3.8.10-0ubuntu1~20.04.9_amd64.deb 404 Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.562 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/python3.8/python3.8-dev_3.8.10-0ubuntu1~20.04.9_amd64.deb 404 Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.562 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.562 Fetched 124 MB in 4s (28.4 MB/s)
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 ERROR: executor failed running [/bin/sh -c apt-get install -y build-essential python3-dev python3-setuptools automake cmake git flex bison libglib2.0-dev libpixman-1-dev cargo libgtk-3-dev ninja-build gcc-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-plugin-dev libstdc++-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-dev]: exit code: 100
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": ------
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": > [8/9] RUN apt-get install -y build-essential python3-dev python3-setuptools automake cmake git flex bison libglib2.0-dev libpixman-1-dev cargo libgtk-3-dev ninja-build gcc-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-plugin-dev libstdc++-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-dev:
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": ------
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": executor failed running [/bin/sh -c apt-get install -y build-essential python3-dev python3-setuptools automake cmake git flex bison libglib2.0-dev libpixman-1-dev cargo libgtk-3-dev ninja-build gcc-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-plugin-dev libstdc++-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-dev]: exit code: 100
Finished Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate"
ERROR
ERROR: build step 2 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1
From what I see it looks like http://archive.ubuntu.com/ goes through fine but http://security.ubuntu.com fails. Strangely it did work for mbed_fuzz_dtlsclient.
From what I see it looks like http://archive.ubuntu.com/ goes through fine but http://security.ubuntu.com fails. Strangely it did work for mbed_fuzz_dtlsclient.
Yep, this should be a flaky internet connection failure. We sometimes see this but cannot do much other than re-run the exp.
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-11-dgfuzz-1 --fuzzers dgfuzz
Experiment 2024-07-11-dgfuzz-1 data and results will be available later at:
The experiment data.
The experiment report(experimental).
Apt is still not playing ball it seems. It's trying to fetch:
http://security.ubuntu.com/ubuntu/pool/main/p/python3.8/libpython3.8_3.8.10-0ubuntu1~20.04.9_amd64.deb
Looking here it seems there is no 20.04.9, but there is 20.04.10 (python3.8-dbg_3.8.10-0ubuntu1~20.04.10_amd64.deb). ~~As to why it is trying to fetch that particular version, I don't know. The particular command that fails is copied from AFL++ builder.Dockerfile, so maybe that will have the same issue?~~
EDIT: it was missing apt-get update; I guess so far I'd got lucky that the base image was new enough that nothing had been replaced yet.
Hi @DonggeLiu , I figured out that the issue was on my end (see comment above). In the meantime I've added a 'control' condition so that I can see how much of the improvement is down to the DFSan guidance. Could you run the following command please:
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-12-dgfuzz --fuzzers dgfuzz dgfuzz_control
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-13-dgfuzz --fuzzers dgfuzz dgfuzz_control
Experiment 2024-07-13-dgfuzz data and results will be available later at:
The experiment data.
The experiment report(experimental).
Hi @DonggeLiu , could you please run the following for me?
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-18-dgfuzz --fuzzers dgfuzz_bb8e11 dgfuzz_fd6cc1
Thanks!
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-18-dgfuzz --fuzzers dgfuzz_bb8e11 dgfuzz_fd6cc1
Experiment 2024-07-18-dgfuzz data and results will be available later at:
The experiment data.
The experiment report(experimental).
Hi @DonggeLiu , any chance you could run the following for me please?
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-23-dgfuzz --fuzzers dgfuzz_4aacb8 dgfuzz_c286e5 dgfuzz_3f8f81 dgfuzz_0e010d dgfuzz_ab0800
Thanks!
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-29-dgfuzz --fuzzers dgfuzz_4aacb8 dgfuzz_c286e5 dgfuzz_3f8f81 dgfuzz_0e010d dgfuzz_ab0800
Experiment 2024-07-29-dgfuzz data and results will be available later at:
The experiment data.
The experiment report(experimental).
Hi @DonggeLiu , I think I may have caused some issues by letting it try to build so many setups that fail to compile. It ended up taking 16+ hours to get to the point that the experiment-folders appeared in the data dir; and even though I can see that there are runs taking place, it still hasn't created a coverage sub-dir (nor the report obviously).
I've gone in and set all the benchmark.yaml files now so that it shouldn't try to build them, any chance you could rerun it for me? (maybe the old run will need cancelling too if it's causing issues)
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-31-dgfuzz --fuzzers dgfuzz_4aacb8 dgfuzz_c286e5 dgfuzz_3f8f81 dgfuzz_0e010d dgfuzz_ab0800
Thanks, Dan