tarpaulin icon indicating copy to clipboard operation
tarpaulin copied to clipboard

Fail decreasing

Open nikgul opened this issue 9 months ago • 3 comments

This pull request implements the --fail-decreasing command line option, which allows tarpaulin to return an error if the coverage percentage has decreased, similar to --fail-under which fails the run if the coverage is under a certain threshold.

Current feature set implemented:

  • If --fail-under is used by it self, it should behave like before
  • If --fail-decreasing is used by it self, it will return an error if the coverage percentage is decreasing.
  • If --fail-under and --fail-decreasing is used together is works as follows, if the coverage is below the threshold, it error out if the coverage is decreasing, if the coverage is above the threshold, the coverage can increase and decrease freely.

I have had a quick look at the tests, but are uncertain how and if I should make a test case for this, becaues it would require multi step tests, a first run, one increasing, one returning the same coverage, and one where it is decreasing.

This fixes: #1411

Note: I'm new to rust, so if you have any changes you want made, there could be some terminoligy, that i'm not fluent in or familier with yet :-)

nikgul avatar Oct 23 '23 12:10 nikgul

I have now implemented test cases for the feature, and while doing so found 2 bugs that I fixed.

During test runs I see random test failures in:

  • cargo-tarpaulin::integration config_file_coverage
  • cargo-tarpaulin::integration follow_exes_down
  • cargo-tarpaulin::integration llvm_sanity_test

Should I take a look at them, and submit a seperate PR, i expect it is something similar to what I had to deal with when I was writing the test cases in this PR, there may be more like it ?

nikgul avatar Oct 25 '23 13:10 nikgul

Just at work so will look properly later, but I use set_profraw_dir on the target to stop tests clobbering each other.

xd009642 avatar Oct 25 '23 13:10 xd009642

I tried that, but couldn't get it to work, ended up using config.set_target_dir() instead, which works nicely.

nikgul avatar Oct 25 '23 13:10 nikgul