forest icon indicating copy to clipboard operation
forest copied to clipboard

Regression testing using `fuzzy` and the archival data

Open lemmih opened this issue 2 years ago • 0 comments

Issue summary

The entire Filecoin blockchain is stored in CloudFlare. With this data, we can do regression testing over arbitrary ranges of epochs.

Regression testing is likely to be too slow for the per-PR CI. Manually triggering the regression tests might be the best option.

The blockchain data covers all of the network versions (currently from NV1 to NV20). Forest has support for NV16 and newer. Epochs lower than 1_960_320 should not be tested.

Tasks:

  • [ ] Create a script that automatically downloads snapshots and validates Forest over a range of epochs. The script should accept filters such as network version and epoch range. How many epochs to validate should also be configurable (default to 30k).
  • [ ] The script should pick a random block of epochs in the given range and validate using forest-tool snapshot validate --check-links 0 --check-stateroots EPOCHS.
  • [ ] Make sure files are properly cleaned-up. Otherwise, the server will run out of disk space.
  • [ ] Trigger the script to run on fuzzy when @regression-check is commented in a PR.
  • [ ] Support arguments such as @regression-check nv18 that limit the testing to a specific network version.
  • [ ] Write results in a comment response.

Other information and links

See Hubert's grpc_bench for inspiration w.r.t. workflows triggered by comments: https://github.com/LesnyRumcajs/grpc_bench/blob/master/.github/workflows/issue_bench.yml

lemmih avatar Oct 11 '23 14:10 lemmih