google
Seed experiment
This is a experiment to see how five fuzzers (afl, aflpp, libfuzzer, honggfuzz, libafl) behaves when supplied with a empty seeds and saturated seed corpus.
First I want do an empty seed experiment. Later i will change it to use ossfuzz corpus
The command is
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty_seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty_seeds-2 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl
@DonggeLiu Could you run the command?
gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-empty_seeds --fuzzers aflplusplus libafl honggfuzz libfuzzer afl aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds
Sure, but there are 2 issues:
- Could you please follow the same date format in experiment names? 2024-08-14. This help us keep the cloud storage more organized.
- Like last time, this experiment requests too many fuzzers. Could you please split them into 2 experiments. Let's keep #fuzzers <= 5 in each exp.
ok i edited and fixed the issues https://github.com/google/fuzzbench/pull/2025#issuecomment-2286274645
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-15-empty_seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-15-empty-seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty-seeds-2 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl
2024-08-14-empty-seeds-2
it looks like it didn't run can you check what is the reason?
I think it is because of the same issue I fixed in #2023 Now that you have updated your branch, let me re-run the exp.
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-empty-seeds-1 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-empty-seeds-2 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds
Experiment 2024-08-17-empty-seeds-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
Experiment 2024-08-17-empty-seeds-2 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
Hi @tokatoka, I started two test exp above. If they work well, could you please rebase to adopt this change? I forgot to revert temp changes in a previous PR.
Thanks!
now it's truly okay (with the green ci) could you run the same command? @DonggeLiu
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-21-empty-seeds-1 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-21-empty-seeds-2 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds
hmm the experiment-data has data in its folder but the report is not generated do you know the reason? @DonggeLiu
I updated libafl commit here too. Can we start the experiment here?
Sure, I wonder if that heisenbug will haunt us here. Let's run for once, and if the bug reappears, I will create a new PR with my debug logs.
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-27-empty-seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds
No it doesn't work. Just like before. The experiment-data is there but the report is missing
@DonggeLiu How can I help to debug this stuff..?
https://github.com/google/fuzzbench/pull/2000#issuecomment-2272406883 Could it be related to this? maybe you can try run just zlib on it's own and see if there's the same error?
@DonggeLiu Can you create a debug branch for this too..? For this one, 0 reports were created
yep I saw a lot of "Fuzz target binary not found." errors:
It's likely the same error again. Weird that it stopped occurring for a year and suddenly came back.
@DonggeLiu How can I help to debug this stuff..?
#2000 (comment) Could it be related to this? maybe you can try run just
zlibon it's own and see if there's the same error?
Thanks for helping @DanBlackwell . We will investigate more in a new PR.
@DonggeLiu Can you create a debug branch for this too..? For this one, 0 reports were created
I will need your help to resolve the conflict before I can rebase and create a new PR : )
Auto-merging fuzzers/libafl/builder.Dockerfile
CONFLICT (content): Merge conflict in fuzzers/libafl/builder.Dockerfile
error: could not apply 8b14d84f... Update LibAFL (#2036)
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Could not apply 8b14d84f... Update LibAFL (#2036)
Or you could cherry-pick my commits from https://github.com/google/fuzzbench/pull/2038.