coveragepy
coveragepy copied to clipboard
nedbat
"source" config combined with pytest-xdist leads to incorrect coverage
Describe the bug Either source behaves weirdly in combination with xdist, or the documentation around how source/include work could be improved.
When supplying certain source/include values in particular ways with xdist, you can either get 0% or 100%.
In my real-life usecase, I actually see 16% coverage reported, where the last xdist process reports some amount of coverage while everything else reports 0. But perhaps due to the simpler test-setup i lay out below, it only seems to result in 0s.
It seems, by description, similar to https://github.com/nedbat/coveragepy/issues/389
To Reproduce How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:
- What version of Python are you using? 3.9.6
- What version of coverage.py shows the problem? 6.3.2
- What versions of what packages do you have installed? pytest-xdist==2.5.0
❯ tree
.
├── poetry.lock
├── pyproject.toml
├── src
│ └── foo
│ ├── __init__.py
│ └── meow.py
└── test.py
# pyproject.toml
[tool.poetry]
name = "foo"
version = "0.0.0"
description = ""
authors = []
packages = [
{ include = "foo", from = "src" },
]
[tool.coverage.report]
show_missing = true
skip_covered = true
[tool.coverage.run]
source = ["src"]
parallel = true
branch = true
# meow.py
def meow():
print("meow")
def meow2():
print("meow")
def meow3():
print("meow")
meow()
# test.py
from foo.meow import meow, meow2, meow3
def test_meow():
meow()
def test_meow2():
meow2()
def test_meow3():
meow3
If you run the following, you get the correct result (86% in this case)
coverage run -m py.test test.py && coverage combine && coverage report
If you run the following, you get 0%
coverage run -m py.test -n 3 test.py && coverage combine && coverage report
Perhaps notable other options I tried:
source = ["src/"] # 0%
source = ["foo"] # 0%
source = ["src/foo"] # 0%
source = ["src/foo"] # 0%
include = ["src"] # 0%
include = ["src/*"] # 86% hurray!
So tl;dr the behavior of source seems to be different with/without xdist, perhaps
in combination given the fact that the package is nested within a src/ directory
instead of foo/ being implicitly on the path.
*Expected behavior
I expect the behavior of the source option to react the same regardless of
use of the -n flag from pytest-xdist.
Alternatively, if this is difficult to work around, just that perhaps the docs
be made more clear that one needs to use include with a pattern.
I can confirm that there is a clear bug in this area and when I commented source and used include the reported coverage went from 12% to 23%, even if xdist run on 10 cores (m1).
This is still far from the real coverage on the project, which is around 84% as measured with pytest-cov.
I should mention that I reached that place while trying to avoid using pytest-cov because it was reporting some warnings about modules being imported too soon. Funny bit is that even with these warnings, the pytest-cov reported far more than coverage.
# pyproject.toml
[tool.coverage.run]
parallel = true
concurrency = ["multiprocessing", "thread"]
I am really curious to find a project using just coverage.py + pytest + xdist, what does properly record the coverage, maybe I could stop what is causing the incomplete coverage.
Another thing that I found quite weird was that I seen the coverage files being exact 102400 in file size.
After spending a good number of hours today trying to make it work w/o pytest-cov, I think I found the trick. In fact it was only one project that managed to get it working and that was via https://github.com/python-attrs/attrs/pull/1011/files change. change the the magic sauce that nobody documented was the installation of a 6 years package named coverage-enable-subprocess -- magically once that package is installed, my coverage magically switched from 23% to 89%.
I want to understand what's going wrong here more, but:
Alternatively, if this is difficult to work around, just that perhaps the docs be made more clear that one needs to use
includewith a pattern.
The help for coverage run says:
--include=PAT1,PAT2,...
Include only files whose paths match one of these
patterns. Accepts shell-style wildcards, which must be
quoted.
The docs for specifying source files say:
The include and omit file name patterns follow typical shell syntax: * matches any number of characters and ? matches a single character. Patterns that start with a wildcard character are used as-is, other patterns are interpreted relative to the current directory...
If you have suggestions for how to make it clearer that include needs a pattern, I'm open to them.
I dont remember writing that 😅, the content in the docs seems clear enough. Instead it might be better to warn in the event that includes are supplied which ultimately result in no matches? But honestly that's more of an unrelated feature request.
More relevant to the bit where i said "Alternatively, if this is difficult to work around", i guess i'd now instead suggest warning if xdist is detected to be used in combination with source; if source cannot be made to work.
Definitely this is because you have to configure coverage.py to measure subprocesses. You can do that a number of ways:
- Use the steps in https://coverage.readthedocs.io/en/6.5.0/subprocess.html
- Use the coverage-enable-subprocess package
- Use pytest-cov
It might be time for coverage.py to have a better way to do this itself...
Not sure if this helps, but on Windows (with coverage, pytest, and pytest-xdist), if I run
coverage run --debug=trace --parallel-mode -m pytest -n auto
I consistently get an OSError: [WinError 6] The handle is invalid. Perhaps there is a race condition when trying to write out the .coverage.* files?
Afterwards, if I run
coverage combine && coverage report -m
I get 0% coverage like everyone else is saying. If I don't use xdist and just run
coverage run --debug=trace -m pytest && coverage combine && coverage report -m
Everything works fine...¯(º_o)/¯
traceback
Traceback (most recent call last): File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pytest\__main__.py", line 5, in
Also unfortunately, adding coverage-enable-subprocess to my project doesn't seem to work on my end :/