pytest-cov icon indicating copy to clipboard operation
pytest-cov copied to clipboard

Published 20 hours ago • verified publisher icon pytest-dev

Inconsistent coverage reports

Open apriha opened this issue 5 years ago • 6 comments

Hi guys, I'm not sure if this is a pytest-cov, coverage, or Travis CI issue, but recently my Travis builds have started producing inconsistent coverage reports. For example, with this job, the coverage report is including src and site-packages:

https://travis-ci.org/apriha/snps/jobs/604796573

However, with this job (same parent build as the previous) the coverage report only includes site-packages:

https://travis-ci.org/apriha/snps/jobs/604796575

Any idea what's happening here? Thanks in advance!

apriha avatar Oct 30 '19 05:10 apriha

Is it always the same test configuration that lacks the src coverage or is it kind of random?

ionelmc avatar Oct 30 '19 11:10 ionelmc

It seems random, and I even saw it once where the src coverage was reported for a job and then not reported after restarting the same job shortly after the first.

Note that this just started happening in the last few weeks from what I can tell...

On Wed, Oct 30, 2019 at 4:17 AM Ionel Cristian Mărieș < [email protected]> wrote:

Is it always the same test configuration that lacks the src coverage or is it kind of random?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pytest-dev/pytest-cov/issues/356?email_source=notifications&email_token=ABNLQAU3ZSOIX4ENVPWSKDDQRFUONA5CNFSM4JGTRD6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECTZFCI#issuecomment-547852937, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNLQASM23GE6F35Z2RIKZTQRFUONANCNFSM4JGTRD6A .

apriha avatar Oct 30 '19 13:10 apriha

Here is an example of a build where src was included in the report, and after restarting the job a few minutes after the first, src was not included:

https://codecov.io/gh/apriha/snps/commit/57c204a4a436def6fe38d6bce610234c7ed92478/build

Specifically, here are the raw reports:

1st report https://codecov.io/codecov/v4/raw/2019-10-23/9E6AA2A1AAF76DC349CF798DB6EB70E2/57c204a4a436def6fe38d6bce610234c7ed92478/43ba98c3-e464-43d2-9fe1-19108ce53974.txt

2nd report (after restarting job) https://codecov.io/codecov/v4/raw/2019-10-23/9E6AA2A1AAF76DC349CF798DB6EB70E2/57c204a4a436def6fe38d6bce610234c7ed92478/363ee630-0128-4c85-89c9-ca3ab38d93da.txt

Notice how the 2nd report doesn't have the element <package branch-rate="0.0229" complexity="0" line-rate="0.02011" name="snps">.

apriha avatar Oct 30 '19 17:10 apriha

Ok, it looks like the coverage that's being reported in the src directory is a function that gets called via Pool.

I then came across this and added the recommended fix (cleanup_on_sigterm), but the src directory is still getting picked up. Any ideas on where to go from here?

apriha avatar Nov 02 '19 05:11 apriha

FYI, since results are not being combined, removing the following from setup.cfg fixed the issue for me:

[coverage:paths]
source =
    src
    */site-packages

Thanks for providing pytest-cov!

apriha avatar Nov 02 '19 06:11 apriha

Is this related to https://github.com/nedbat/coveragepy/issues/883#issuecomment-650562896?

JulienPalard avatar Jun 27 '20 17:06 JulienPalard