Add script to generate fuzz coverage
This script generates the fuzz coverage in a html file when run .
Considerations
- The script assumes a
corpusdirectory infuzzdirectory. - Also assumes the corpus for a particular target to be under the name of the target itself. for eg - corpus for
base32_target.rswould be undercorpus/base32_target. - The corpus I worked with was generated using
libFuzzerit may/may not work with corpus generated from other fuzzers.
How to run
# from the root directory
chmod +x contrib/generate_fuzz_coverage.sh
contrib/generate_fuzz_coverage.sh
👋 I see @jkczyz was un-assigned. If you'd like another reviewer assignemnt, please click here.
If needed I can also push my script to generate corpus from libFuzzer that create the corpus according to the considerations if the reviewers deem it worthy .
👋 The first review has been submitted!
Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.
I wonder if we should include this coverage data in our codecov output in PRs...On the one hand its not "reliable" coverage data in that we aren't actually testing the behavior, but on the other hand we are at least hitting the code lines and presumable would find crashes. WDYT?
I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.
I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.
One reason to do this would be to only include "trivially-reachable" things in the codecov output. We don't really want to include stuff as coverage if the fuzzer happened to reach it only once, but if the fuzzer found its way into it in a minute of CI time, its probably decent coverage :)
I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.
One reason to do this would be to only include "trivially-reachable" things in the codecov output. We don't really want to include stuff as coverage if the fuzzer happened to reach it only once, but if the fuzzer found its way into it in a minute of CI time, its probably decent coverage :)
Makes sense if thats the case.
Do you intend to go ahead and run this in CI? Or would you prefer not to?
added this script to run in CI .
1.63 versions are failing because llvm-cov requires 1.65 , other 2 failures are because of storage issue on the CI instance.
Failure caused because of no storage left on the ci infra
Working locally when run with the CI workflow.
CI passed . had to nuke the entire target directory
Might be worth giving CI a kick now that we fixed the No space left on device issue (for now) with #3916.