rust-lightning icon indicating copy to clipboard operation
rust-lightning copied to clipboard

Add script to generate fuzz coverage

Open Prabhat1308 opened this issue 8 months ago • 4 comments

This script generates the fuzz coverage in a html file when run .

Considerations

  1. The script assumes a corpus directory in fuzz directory.
  2. Also assumes the corpus for a particular target to be under the name of the target itself. for eg - corpus for base32_target.rs would be under corpus/base32_target.
  3. The corpus I worked with was generated using libFuzzer it may/may not work with corpus generated from other fuzzers.

How to run

# from the root directory
chmod +x contrib/generate_fuzz_coverage.sh
contrib/generate_fuzz_coverage.sh

Prabhat1308 avatar Apr 08 '25 19:04 Prabhat1308

👋 I see @jkczyz was un-assigned. If you'd like another reviewer assignemnt, please click here.

ldk-reviews-bot avatar Apr 08 '25 19:04 ldk-reviews-bot

If needed I can also push my script to generate corpus from libFuzzer that create the corpus according to the considerations if the reviewers deem it worthy .

Prabhat1308 avatar Apr 08 '25 19:04 Prabhat1308

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

ldk-reviews-bot avatar Apr 09 '25 14:04 ldk-reviews-bot

I wonder if we should include this coverage data in our codecov output in PRs...On the one hand its not "reliable" coverage data in that we aren't actually testing the behavior, but on the other hand we are at least hitting the code lines and presumable would find crashes. WDYT?

I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.

Prabhat1308 avatar Apr 09 '25 15:04 Prabhat1308

I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.

One reason to do this would be to only include "trivially-reachable" things in the codecov output. We don't really want to include stuff as coverage if the fuzzer happened to reach it only once, but if the fuzzer found its way into it in a minute of CI time, its probably decent coverage :)

TheBlueMatt avatar May 14 '25 15:05 TheBlueMatt

I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.

One reason to do this would be to only include "trivially-reachable" things in the codecov output. We don't really want to include stuff as coverage if the fuzzer happened to reach it only once, but if the fuzzer found its way into it in a minute of CI time, its probably decent coverage :)

Makes sense if thats the case.

Prabhat1308 avatar May 18 '25 21:05 Prabhat1308

Do you intend to go ahead and run this in CI? Or would you prefer not to?

TheBlueMatt avatar Jun 12 '25 18:06 TheBlueMatt

added this script to run in CI .

Prabhat1308 avatar Jun 13 '25 09:06 Prabhat1308

1.63 versions are failing because llvm-cov requires 1.65 , other 2 failures are because of storage issue on the CI instance.

Prabhat1308 avatar Jun 15 '25 19:06 Prabhat1308

Failure caused because of no storage left on the ci infra

Prabhat1308 avatar Jul 01 '25 15:07 Prabhat1308

Working locally when run with the CI workflow.

Screenshot 2025-07-08 at 9 52 01 PM

Prabhat1308 avatar Jul 08 '25 16:07 Prabhat1308

CI passed . had to nuke the entire target directory

Prabhat1308 avatar Jul 08 '25 17:07 Prabhat1308

Might be worth giving CI a kick now that we fixed the No space left on device issue (for now) with #3916.

tnull avatar Jul 10 '25 09:07 tnull