fuzzbench
fuzzbench copied to clipboard
google
Orchesctra: A Collaborative Fuzzer Framework
#1859
Hello dongge, @Alan32Liu I've improved my fuzzer framework based on the data from the last experiment, and it's time to reopen this experiment. Other than that, to avoid duplicating the experiment overhead, I've only submitted the necessary fuzzers, so please review it and get it running.
Best regards, Taotao.
OK, just to be extra safe, would it be alright if we split them into the following two groups and run them separately? Group 1:
- orchestra_236
- orchestra_339
- orchestra_4310
- orchestra_219
- orchestra_116
Group 2:
- orchestra_aflplusplus
- orchestra_honggfuzz
- orchestra_libfuzzer
- pastis
If you'd like to see their result in one table, we can download the raw CSV data (data.csv.gz) from the end of the report, which can be used to regenerate the combined report with selected fuzzers and benchmarks.
If you agree, the commands are: (future note for myself)
gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-1 --fuzzers orchestra_236 orchestra_339 orchestra_4310 orchestra_219 orchestra_116
gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-2 --fuzzers orchestra_aflplusplus orchestra_honggfuzz orchestra_libfuzzer pastis
BTW, just a minor future request if it is not too much trouble:
Instead of editing service/experiment-requests.yaml, would there be any chance that I could invite you to write the experiment request in this format?
You can swap the --experiment-name and --fuzzers parameters with your values.
I would really appreciate that : )
The file service/experiment-requests.yaml was no longer used to trigger experiments, as it requires merging PRs.
The new way (via gcbrun comments) allows us to run experiments and modify code multiple times on the same PR for better maintenance. I believe this improvement will benefit both of us.
My comment above is another example of this : )
I totally agree with you, dongge. 😄
---Original--- From: "Dongge @.> Date: Tue, Sep 26, 2023 08:31 AM To: @.>; Cc: "taotao @.@.>; Subject: Re: [google/fuzzbench] Orchesctra: A Collaborative Fuzzer Framework(PR #1904)
OK, just to be extra safe, would it be alright if we split them into the following two groups and run them separately? Group 1:
- orchestra_236 - orchestra_339 - orchestra_4310 - orchestra_219 - orchestra_116
Group 2: - orchestra_aflplusplus - orchestra_honggfuzz - orchestra_libfuzzer - pastis
If you'd like to see their result in one table, we can download the raw CSV data (data.csv.gz) from the end of the report, which can be used to regenerate the combined report with selected fuzzers and benchmarks.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>
gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-1 --fuzzers orchestra_236 orchestra_339 orchestra_4310 orchestra_219 orchestra_116
gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-2 --fuzzers orchestra_aflplusplus orchestra_honggfuzz orchestra_libfuzzer pastis
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-1 --fuzzers orchestra_236 orchestra_339 orchestra_4310 orchestra_219 orchestra_116
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-27-orchestra-1 --fuzzers orchestra_116 orchestra_216 orchestra_236 orchestra_339 orchestra_4310
Experiment 2023-09-27-orchestra-1 data and results will be available later at:
The experiment data.
The experiment report.
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-2 --fuzzers orchestra_aflplusplus orchestra_honggfuzz orchestra_libfuzzer pastis
Experiment 2023-09-25-orchestra-2 data and results will be available later at:
The experiment data.
The experiment report.
gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-1 --fuzzers orchestra_236 orchestra_339 orchestra_4310 orchestra_219 orchestra_116
I reckon orchestra_219 is actually orchestra_216?
Yes, dongge. The config of 216 is same as 219 when the total time budget less than 23h.
---Original--- From: "Dongge @.> Date: Wed, Sep 27, 2023 10:41 AM To: @.>; Cc: "taotao @.@.>; Subject: Re: [google/fuzzbench] Orchesctra: A Collaborative Fuzzer Framework(PR #1904)
gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-09-25-orchestra-1 --fuzzers orchestra_236 orchestra_339 orchestra_4310 orchestra_219 orchestra_116
I reckon orchestra_219 is actually orchestra_216?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>
Yes, dongge. The config of 216 is same as 219 when the total time budget less than 23h.
I meant there isn't a 219, is this intended?
Yes, you can delete 219, thanks.
---Original--- From: "Dongge @.> Date: Wed, Sep 27, 2023 18:34 PM To: @.>; Cc: "taotao @.@.>; Subject: Re: [google/fuzzbench] Orchesctra: A Collaborative Fuzzer Framework(PR #1904)
Yes, dongge. The config of 216 is same as 219 when the total time budget less than 23h.
I meant there isn't a 219, is this intended?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>