requests icon indicating copy to clipboard operation
requests copied to clipboard

Measure and report test coverage

Open sigmavirus24 opened this issue 1 year ago • 4 comments

This runs pytest within coverage, enables pytest-xdist to improve test run times, and reports the coverage on GitHub Actions for us.

This also ensures that the parallel mode coverage files are ignored and adds things to pytest configuration to make tox command simpler

sigmavirus24 avatar Feb 24 '24 15:02 sigmavirus24

I haven't had a chance to dig into the failure yet, but it's showing 26% coverage which seems extremely low. I tried to go back to find older examples of when codecov was running and am seeing closer to 89% coverage. Do we possibly have something misconfigured?

nateprewitt avatar Feb 26 '24 18:02 nateprewitt

Running in tox I get north of 80% as well but we're not running in tox on GitHub actions so that may be part of the problem

sigmavirus24 avatar Feb 26 '24 22:02 sigmavirus24

It looks like we're possibly hitting an issue with coverage combine similar to what's noted in https://github.com/nedbat/coveragepy/issues/1036.

We're generating 21 files for the 21 tests across 7 interpreters. From Ned's response it sounds like those combinations have subtle variances that may result in false flags for missing coverage? When running the tests locally without parallel=True and generating a single file it shows 87% like I'd expect. If I use parallel = True and don't use combine, we get the same results we're seeing in CI as if we're missing coverage data.

Have you used this workflow in other projects? If not, we might consider dividing coverage by interpreter or another partitioning strategy.

nateprewitt avatar Mar 01 '24 21:03 nateprewitt

I mostly stole this from urllib3 and https://hynek.me/articles/ditch-codecov-python/ but maybe I missed something important

sigmavirus24 avatar Mar 02 '24 14:03 sigmavirus24