google
Archive coverage data alongside corpus archives (from AFL++ fork)
Supercedes #2020. Moving so we (AFL++ people) can collaborate on this PR.
From the original:
Currently, only corpora are saved in the archive and the summaries of coverage are provided at the end of the experiment. This change simply incorporates the saving of the coverage data snapshots next to the trial corpus snapshots.
@DonggeLiu Can we try to do a baseline experiment with this PR again? :slightly_smiling_face: It is fully rebased to the latest changes.
I will integrate the analysis changes once there is a public baseline to point the analysis example at.
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-2028-bases-1 --fuzzers afl aflplusplus libafl libfuzzer
Hi @addisoncrump,I started a test exp above.
Experiment 2024-08-17-2028-bases-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
If it works well and you'd like to run a full exp (23 hours), could you please rebase to adopt this change? I forgot to revert temp changes in a previous PR.
Thanks!
Rebased. The experiment looks good, all the coverage samples were archived.
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-19-2028-bases-1 --fuzzers afl aflplusplus libafl libfuzzer
Experiment 2024-08-19-2028-bases-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
This is a really strange, because 2024-08-19-2028-bases-1 has a list of errors about merging coverage summary:
But 2024-08-17-2028-bases-1 did not have any:
QQ: Is the only thing change between those 2 experiments?
BTW, I noticed this runtime crash in libafl. I don't think it could cause the failure, but it might be interesting to you: https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-08-19-2028-bases-1/experiment-folders/libxml2_xml-libafl/trial-3070882/results/
It did not happen in 2024-08-17-2028-bases-1, maybe because that experiment was very short?
A possible theory: libafl saved some input into its corpus during this crash, which caused measurement failure?
I confirmed that the only difference is that commit, yes.
Let's add some more debugging and run a very short run with all the benchmarks, I guess?
A possible theory: libafl saved some input into its corpus during this crash, which caused measurement failure?
but it should not affect other fuzzers such as aflplusplus runs right?
@tokatoka random libafl crash 🙃
can you reproduce? i used the same setup on fuzzbench but cannot reproduce
I updated. @DonggeLiu Could you run the same command again to see if it fixes the problem or not?
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-22-2028-bases-1 --fuzzers libafl
A possible theory: libafl saved some input into its corpus during this crash, which caused measurement failure?
Also running an experiment without libafl to help verify this theory.
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-22-2028-bases-2 --fuzzers afl aflplusplus libfuzzer
Experiment 2024-08-22-2028-bases-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
Experiment 2024-08-22-2028-bases-2 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
bases-1 seems to be working fine, but bases-2 is not hitting the measurer still.
btw for base-1 it seems all the fuzzers are stuck after 10:45m so it was not a successful run either...
Ops there were a DB issue yesterday which affected both experiments. Let me re-run them
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-23-2028-libafl --fuzzers libafl
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-23-2028-bases --fuzzers afl aflplusplus libfuzzer
we have experiment-folder but not report. so the measurement is still broken
For the report: 2024-08-23-2028-libafl and 2024-08-23-2028-bases is missing but 2024-08-23-2036-bases-1 (from the other PR) is there
For the experiment-data: nothing is missing.
Btw if the experiment on my branch and https://www.fuzzbench.com/reports/experimental/2024-08-23-dgfuzz/index.html ← this experiment are working. would it be possible that the changes in this PR caused the measurement failure??
Experiment 2024-08-23-2028-libafl data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
Experiment 2024-08-23-2028-bases data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).
@addisoncrump would this happen to be related to this PR?
It could be due to this error:
I think the gsutil rm error is at least benign, because @tokatoka shows 2024-08-23-2036-bases-1 can generate a report:
Btw if the experiment on my branch and https://www.fuzzbench.com/reports/experimental/2024-08-23-dgfuzz/index.html ← this experiment are working. would it be possible that the changes in this PR caused the measurement failure??
and it also has gsutil rm error, but not llvm-profdata error:
There was another build error (discussed in #2038, as shown above), but I am sure that one is benign and unrelated to the missing report in the experiment.