forest
forest copied to clipboard
Regression testing using `fuzzy` and the archival data
Issue summary
The entire Filecoin blockchain is stored in CloudFlare. With this data, we can do regression testing over arbitrary ranges of epochs.
Regression testing is likely to be too slow for the per-PR CI. Manually triggering the regression tests might be the best option.
The blockchain data covers all of the network versions (currently from NV1 to NV20). Forest has support for NV16 and newer. Epochs lower than 1_960_320 should not be tested.
Tasks:
- [ ] Create a script that automatically downloads snapshots and validates Forest over a range of epochs. The script should accept filters such as network version and epoch range. How many epochs to validate should also be configurable (default to 30k).
- [ ] The script should pick a random block of epochs in the given range and validate using
forest-tool snapshot validate --check-links 0 --check-stateroots EPOCHS. - [ ] Make sure files are properly cleaned-up. Otherwise, the server will run out of disk space.
- [ ] Trigger the script to run on
fuzzywhen@regression-checkis commented in a PR. - [ ] Support arguments such as
@regression-check nv18that limit the testing to a specific network version. - [ ] Write results in a comment response.
Other information and links
See Hubert's grpc_bench for inspiration w.r.t. workflows triggered by comments: https://github.com/LesnyRumcajs/grpc_bench/blob/master/.github/workflows/issue_bench.yml