Fail coverage based on absolute number of uncovered lines

[jml]

Is your feature request related to a problem? Please describe. When improving the test coverage of a legacy code base, I would like to have a ratchet based on the absolute number of uncovered lines.

There are two aspects to this:

1. Absolute number of uncovered lines

I want test coverage because untested code is very likely to be buggy. The number of bugs is not a function of the percentage of uncovered code, but rather the amount of uncovered code. Saying I've got 98% coverage might impress my friends, but if that means I've got 20kLoC of untested code then it's somewhat meaningless.

More seriously, deleting tested code reduces the test coverage percentage even though coverage is not objectively worse. Working around this takes all sorts of hijinx that are unnecessary when using absolute numbers.

2. Ratchet

This means I would like coverage to fail not only if the coverage has got worse but also if it has got better. Failing if it's worse is obvious—I don't want coverage to get worse. Failing if it's better means that contributors will have to update the number of uncovered lines in the build scripts to lock in the newer, lower number.

Neither of these are a problem with greenfields projects, because then I start with --fail-under=100 immediately, which means I don't need a ratchet, and I have an absolute number of uncovered lines: zero.

Describe the solution you'd like A new option to coverage report called --num-uncovered-lines=N that would cause it to exit with status 2 if there were more than N uncovered lines, and exit with status 3 if there were fewer than N uncovered lines.

The error message should clearly indicate whether coverage got worse or better. In the case where it's better, it should emit the new number of uncovered lines so it is easy to update build machinery.

Describe alternatives you've considered If the ratchet aspect is distasteful, having --max-uncovered-lines=N would also be very useful.

Additional context Here's the script that I'm using for this right now: https://gist.github.com/jml/27d21c9322d112a3b9b5e94508c93928

I run it in CI with something like:

# The number of lines currently without test coverage.
# Do not approve a PR that increases this number.
UNCOVERED_LINES = 761

coverage run --branch -m pytest -q tests/
coverage report --show-missing --skip-covered
python scripts/missing_coverage_lines.py --allowed-missing=${UNCOVERED_LINES} .coverage

[nedbat]

I understand your logic, but this is starting to get into very specific policies. Maybe the best way to support you use is to give you an easy way to get all the statistics in a machine-readable form? For example, #720 suggests a "json report" would would let you get at all the details.

[Zac-HD]

I'd also find both --allowed-missing and --max-missing arguments extremely useful! I've never found percentages less than 100 useful as a target, but it's hard to get there in a single step otherwise. Concretely, most of my projects (e.g. Hypothesis) already enforce 100% coverage; but I found this issue on the way to ask if you'd accept just such an argument because I would like to use --max-missing to drive Pytest's coverage up - at the moment there's just too much noise as unrelated code is added, changed, and removed.

IMO "the amount of uncovered code should be at most X" is actually less specific than "the percentage of uncovered code should be at most X", because it doesn't have to account for the effect of covered code.

In the end it's your call though, and it would be possible to build this from a nice machine-readable format. I'd also be happy to implement these options in a PR, if you'd accept them!

[nedbat]

Maybe this is something to add to a separate coverage-goals checker? https://nedbatchelder.com/blog/202111/coverage_goals.html