pytest-cov
pytest-cov copied to clipboard
pytest-dev
Coverage Report is incorrect when there are multiple modules with same name but in different namespaces
platform win32 -- Python 2.7.16, pytest-4.4.1, py-1.8.0, pluggy-0.11.0 -- c:\workspace\projects\pytest-dummy\.venv\scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\workspace\projects\pytest-dummy
plugins: cov-2.7.1
This issue appears when there are some modules with the same name but in different namespaces within the same project or if you have code in init files. I have this dummy project with the following structure:
| /my-project/
|----| /Module1/
|----|-----| /tests/
|----|-----| __init__.py
|----|-----| MyFunctionality.py
|----| /Module2/
|----|-----| /tests/
|----|-----| __init__.py
|----|-----| MyFunctionality.py
|----| /Module3/
|----|-----| /tests/
|----|-----| __init__.py
|----|-----| AnotherFunctionality.py
If I run the following command:
pytest -v --junitxml unittests.xml --cov=Module1 --cov=Module2 --cov=Module3 --cov-config .coveragerc --cov-report xml --cov-report term
Although the "term" Cobertura report shows correctly the coverage of all the modules:
Name Stmts Miss Cover
-----------------------------------------------------
Module1\MyFunctionality.py 5 5 0%
Module1\__init__.py 0 0 100%
Module2\MyFunctionality.py 5 5 0%
Module2\__init__.py 0 0 100%
Module3\AnotherFunctionality.py 5 5 0%
Module3\__init__.py 0 0 100%
-----------------------------------------------------
TOTAL 15 15 0%
Coverage XML written to file coverage.xml
The coverage.xml file includes only the coverage for two files:
<?xml version="1.0" ?>
<coverage branch-rate="0" branches-covered="0" branches-valid="0" complexity="0" line-rate="0" lines-covered="0" lines-valid="15" timestamp="1557386168872" version="4.5.3">
<!-- Generated by coverage.py: https://coverage.readthedocs.io -->
<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
<sources>
<source>C:\workspace\projects\pytest-dummy\Module1</source>
<source>C:\workspace\projects\pytest-dummy\Module2</source>
<source>C:\workspace\projects\pytest-dummy\Module3</source>
</sources>
<packages>
<package branch-rate="0" complexity="0" line-rate="0" name=".">
<classes>
<class branch-rate="0" complexity="0" filename="AnotherFunctionality.py" line-rate="0" name="AnotherFunctionality.py">
<methods/>
<lines>
<line hits="0" number="1"/>
<line hits="0" number="2"/>
<line hits="0" number="3"/>
<line hits="0" number="5"/>
<line hits="0" number="6"/>
</lines>
</class>
<class branch-rate="0" complexity="0" filename="MyFunctionality.py" line-rate="0" name="MyFunctionality.py">
<methods/>
<lines>
<line hits="0" number="1"/>
<line hits="0" number="2"/>
<line hits="0" number="3"/>
<line hits="0" number="5"/>
<line hits="0" number="6"/>
</lines>
</class>
<class branch-rate="0" complexity="0" filename="__init__.py" line-rate="1" name="__init__.py">
<methods/>
<lines/>
</class>
</classes>
</package>
</packages>
</coverage>
As you can see, the report only shows one "Functionality.py" coverage and on "init.py" where it should show all of them. If you try to parse this log with Jenkins using the Cobertura plugin, it will only show one file for every group of files that share name (and only one init file as well).
This issue was not reproduced in version pytest-cov==2.2.0, where it would treat the coverage of each file separately in the xml report even if the names where the same.
Can you make a reproducer? I can't tell what the problem is from just this. It looks like in the xml the packages were merged and the root disappeared. Maybe @nedbat has seen this problem before but I haven't and need a reproducer example.
@ionelmc here you have attached a project that will cause this behavior. Just download it and execute the following command with pytest-4.4.1 and pytest-cov-2.7.1:
pytest -v --junitxml unittests.xml --cov=Module1 --cov=Module2 --cov=Module3 --cov-config .coveragerc --cov-report xml --cov-report term
Just an idea/remark: try to use include = Module1, Module2, Module3 in the coverage config, and specify --cov only (without sources).
Also try to figure out how coverage.py itself handles it, e.g. look at coverage report afterwards, and try using coverage run -m pytest ….
Hello! I face same trouble (in CI pipeline between atlassian bamboo and sonarqube) and spent many hours for debug. Finally i googled this issue, but as i can understand, here is no clear explanation. So. Finally i found out following: if you have similar project structure (like described in issue), use whole folder, not apps/modules. For example, if you have module1, module2, module3 and use "--cov module1 --cov module2 --cov module3", just replace it with "--cov ." and add some omit/restictions to skip necessary files. This helps me (sonarqube with previous coverage report show coverage at 46% for our project, now it shows real 97% (as you understands, we often repeat names in different packages - like helpers.py and other))
Explanation: as i understands (i may misunderstood logic), coverage get files from folders ("--cov module1 --cov module2 --cov module3") and use paths as keys, so if you have same names, they obiously will be overriden in final report file, because it's just flat list, without parent folders (parent folder can be found only at the top of report file (cubertura at least)).
It seems very obvious for expirienced users of coverage, but i was not like them (though i spent many years in it industry and many years i generated those reports, but faced this issue first time), so i decided to save time for any user who gets here
