subtensor icon indicating copy to clipboard operation
subtensor copied to clipboard

Code coverage suite (rebased #848 with workspace restructure)

Open ales-otf opened this issue 11 months ago • 3 comments

Description

Adds code coverage suite to the repo

  • [x] find and parse all pallets dynamically and in parallel
  • [x] code to import missing sections of pallets defined with #[import_section(..)] / #[pallet_section]
  • [x] collect metadata on all extrinsics/calls in all pallets, in parallel
  • [ ] collect metadata on free functions defined within pallets (non-calls)
  • [x] find all tests in the repo in parallel
  • [x] find all benchmark tests (v2 syntax) in the repo in parallel
  • [x] upgrade existing benchmarking v1 tests to v2 (only one file, quick)
  • [x] add a quick little lint disallowing benchmarking v1 syntax and enable this lint in build.rs
  • [ ] collect metadata and coverage info for hooks, might be impractical, happy path would be if people commonly enter hooks manually within tests, then you can detect tests that do this and expect one covering each hook state, possibly for each pallet
  • [x] analyze tests and generate coverage statistics based on what functions are/are not mentioned across all tests
  • [x] display code coverage results in the terminal from the build script (using sam's build-print crate)
  • [ ] by default don't run code coverage unless RUN_COVERAGE env var is present (from within build.rs)

This PR also restructures the workspace by removing the root library and encapsulating logic from the root's build.rs into linting crate API. It adds build.rs scripts to run the lints independently for each crate.

The code-coverage is a binary, which you run using:

cargo run -p subtensor-code-coverage

The default crate in workspace is configured to be node-subtensor, so all commands go to this crate, if you don't pass -p <name> argument. It allows cargo run node, without specifying the package name.

Future work:

  • take function complexity into account (number of distinct terminal points = needs more coverage)
  • use conventional code coverage tools for everything that isn't a pallet and then roll this information into the final report
  • once 100% coverage is achieved, have CI fail if coverage drops below 100% and flag the specific functions/extrinsics/whatever that are untested
  • take function call graph into account -- if function or extrinsic A calls function or extrinsic B, then coverage on A at least partially covers B

Related Issue(s)

  • Closes #[issue number]

Type of Change

  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [X] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] Documentation update
  • [ ] Other (please describe):

Checklist

  • [ ] I have performed a self-review of my own code
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have run cargo fmt and cargo clippy to ensure my code is formatted and linted correctly
  • [ ] I have made corresponding changes to the documentation
  • [ ] My changes generate no new warnings
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] New and existing unit tests pass locally with my changes
  • [ ] Any dependent changes have been merged and published in downstream modules

ales-otf avatar Nov 20 '24 21:11 ales-otf