subtensor
subtensor copied to clipboard
Code coverage suite (rebased #848 with workspace restructure)
Description
Adds code coverage suite to the repo
- [x] find and parse all pallets dynamically and in parallel
- [x] code to import missing sections of pallets defined with
#[import_section(..)]/#[pallet_section] - [x] collect metadata on all extrinsics/calls in all pallets, in parallel
- [ ] collect metadata on free functions defined within pallets (non-calls)
- [x] find all tests in the repo in parallel
- [x] find all benchmark tests (v2 syntax) in the repo in parallel
- [x] upgrade existing benchmarking v1 tests to v2 (only one file, quick)
- [x] add a quick little lint disallowing benchmarking v1 syntax and enable this lint in
build.rs - [ ] collect metadata and coverage info for hooks, might be impractical, happy path would be if people commonly enter hooks manually within tests, then you can detect tests that do this and expect one covering each hook state, possibly for each pallet
- [x] analyze tests and generate coverage statistics based on what functions are/are not mentioned across all tests
- [x] display code coverage results in the terminal from the build script (using sam's
build-printcrate) - [ ] by default don't run code coverage unless
RUN_COVERAGEenv var is present (from withinbuild.rs)
This PR also restructures the workspace by removing the root library and encapsulating logic from the root's build.rs into linting crate API. It adds build.rs scripts to run the lints independently for each crate.
The code-coverage is a binary, which you run using:
cargo run -p subtensor-code-coverage
The default crate in workspace is configured to be node-subtensor, so all commands go to this crate, if you don't pass -p <name> argument. It allows cargo run node, without specifying the package name.
Future work:
- take function complexity into account (number of distinct terminal points = needs more coverage)
- use conventional code coverage tools for everything that isn't a pallet and then roll this information into the final report
- once 100% coverage is achieved, have CI fail if coverage drops below 100% and flag the specific functions/extrinsics/whatever that are untested
- take function call graph into account -- if function or extrinsic A calls function or extrinsic B, then coverage on A at least partially covers B
Related Issue(s)
- Closes #[issue number]
Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation update
- [ ] Other (please describe):
Checklist
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have run
cargo fmtandcargo clippyto ensure my code is formatted and linted correctly - [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream modules