stacks-core icon indicating copy to clipboard operation
stacks-core copied to clipboard

Mutation testing - logger tracking missing/timeout/unviable tests

Open ASuciuX opened this issue 1 year ago • 3 comments

Applicable issues

  • fixes #3806
  • #4177

Additional info (benefits, drawbacks, caveats)

Solutions and Recommended Usage

Cargo-mutant does an incremental build for each mutation, that’s why this solution comes with the implementation of tracking the output and only updating it with the differences, to run 10-20 mutants instead of 4000.

CI YML File: Are linked with the actions in the actions' repo https://github.com/stacks-network/actions/pull/3.

Should be executed with every PR, as it tests only the updated or newly added functions in the PR's commits.

Handling Exit Codes (docs)

In the logger workflow they are updating the place they belong to.

Checklist

  • [x] Fix mutation testing on packages to run on the supposed functions, not just on some
  • [x] PR CI checker to have all functions tested
    • missed/unviable/timeout outputs result in an error and stop the PR from being merged
    • outputs the functions that have to be tested
  • [x] On merge on main/develop/next branch override (by appending) current output with the merged one
  • [x] Mutation Testing Coverage for the already existing functions
    • [ ] Mutation testing triggering on git diff that didn't affect the functions, but affected the tests. (if the function already exists and is not modified and the tests that use it are created/updated, the functions will not be included in the cargo mutants)
  • [ ] Mutants initial sync completed for develop and next branches

ASuciuX avatar Dec 15 '23 14:12 ASuciuX

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (1f5cc3f) 62.42% compared to head (53cc9c4) 82.61%. Report is 20 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff              @@
##           develop    #4179       +/-   ##
============================================
+ Coverage    62.42%   82.61%   +20.18%     
============================================
  Files          401      401               
  Lines       286383   288405     +2022     
============================================
+ Hits        178775   238256    +59481     
+ Misses      107608    50149    -57459     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Dec 15 '23 14:12 codecov[bot]

Feedback from PR meeting - let's move to draft for now.

obycode avatar Dec 19 '23 15:12 obycode

Related discussion in Nakamoto slack channel.

obycode avatar Dec 19 '23 15:12 obycode