coveragepy
coveragepy copied to clipboard
Reports using multiple `[run]` `source=` entries can lose data from colliding top level files
Describe the bug
- When using
[run]source=with multiple values at least top level files can collide resulting in missing data in a report such ascoverage.xml. Specifically in my case, mypackage/__init__.pyandtests/__init__.pycollide. This behavior feels more like what I would expect from[paths]source=. - Comparing
[run]source=tosource_pkgs=with multiple values shows more differences than I would expect from the docs. This fixes 1), only a single<source>entry is present which is empty (was not empty with 6.5.0) in thecoverage.xml. With 6.5.0, that single entry was an absolute path instead of a relative path.
If these two things both seem like issue but my judgement that they are the same thing is wrong, certainly we can relocate 2) to a separate issue. Also, the side note of the empty <source></source> tag in the XML output might be another issue, I only just now noticed it when running against master to create this report.
Note in the reports below how coverage.source.xml only has results for the tests/__init__.py file and not for package/__init__.py.
https://github.com/altendky/example001/blob/8a76e61a32c77a1ce00699d06dc79d00e4ae9f7d/coverage.no_source.xml
Click to see `coverage.no_source.xml`
<?xml version="1.0" ?>
<coverage version="7.0.0b2.dev1" timestamp="1671205057420" lines-valid="9" lines-covered="9" line-rate="1" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.0.0b2.dev1 -->
<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
<sources>
<source></source>
</sources>
<packages>
<package name="package" line-rate="1" branch-rate="0" complexity="0">
<classes>
<class name="__init__.py" filename="package/__init__.py" complexity="0" line-rate="1" branch-rate="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="4" hits="1"/>
<line number="5" hits="1"/>
</lines>
</class>
</classes>
</package>
<package name="tests" line-rate="1" branch-rate="0" complexity="0">
<classes>
<class name="__init__.py" filename="tests/__init__.py" complexity="0" line-rate="1" branch-rate="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="3" hits="1"/>
<line number="4" hits="1"/>
<line number="6" hits="1"/>
<line number="7" hits="1"/>
</lines>
</class>
<class name="__main__.py" filename="tests/__main__.py" complexity="0" line-rate="1" branch-rate="0">
<methods/>
<lines/>
</class>
</classes>
</package>
</packages>
</coverage>
https://github.com/altendky/example001/blob/8a76e61a32c77a1ce00699d06dc79d00e4ae9f7d/coverage.source.xml
Click to see `coverage.source.xml`
<?xml version="1.0" ?>
<coverage version="7.0.0b2.dev1" timestamp="1671205057646" lines-valid="9" lines-covered="9" line-rate="1" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.0.0b2.dev1 -->
<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
<sources>
<source>package</source>
<source>tests</source>
</sources>
<packages>
<package name="." line-rate="1" branch-rate="0" complexity="0">
<classes>
<class name="__init__.py" filename="__init__.py" complexity="0" line-rate="1" branch-rate="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="3" hits="1"/>
<line number="4" hits="1"/>
<line number="6" hits="1"/>
<line number="7" hits="1"/>
</lines>
</class>
<class name="__main__.py" filename="__main__.py" complexity="0" line-rate="1" branch-rate="0">
<methods/>
<lines/>
</class>
</classes>
</package>
</packages>
</coverage>
https://github.com/altendky/example001/blob/8a76e61a32c77a1ce00699d06dc79d00e4ae9f7d/coverage.source_pkgs.xml
Click to see `coverage.source_pkgs.xml`
<?xml version="1.0" ?>
<coverage version="7.0.0b2.dev1" timestamp="1671205057874" lines-valid="9" lines-covered="9" line-rate="1" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.0.0b2.dev1 -->
<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
<sources>
<source></source>
</sources>
<packages>
<package name="package" line-rate="1" branch-rate="0" complexity="0">
<classes>
<class name="__init__.py" filename="package/__init__.py" complexity="0" line-rate="1" branch-rate="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="2" hits="1"/>
<line number="4" hits="1"/>
<line number="5" hits="1"/>
</lines>
</class>
</classes>
</package>
<package name="tests" line-rate="1" branch-rate="0" complexity="0">
<classes>
<class name="__init__.py" filename="tests/__init__.py" complexity="0" line-rate="1" branch-rate="0">
<methods/>
<lines>
<line number="1" hits="1"/>
<line number="3" hits="1"/>
<line number="4" hits="1"/>
<line number="6" hits="1"/>
<line number="7" hits="1"/>
</lines>
</class>
<class name="__main__.py" filename="tests/__main__.py" complexity="0" line-rate="1" branch-rate="0">
<methods/>
<lines/>
</class>
</classes>
</package>
</packages>
</coverage>
To Reproduce How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:
- What version of Python are you using?
- CPython 3.10 from pyenv in Ubuntu Linux
Python 3.10.8 (main, Oct 15 2022, 17:48:51) [GCC 9.4.0]
- What version of coverage.py shows the problem? The output of
coverage debug sysis helpful.
-- sys -------------------------------------------------------
coverage_version: 7.0.0b2.dev1
coverage_module: /home/altendky/tmp/e/venv/lib/python3.10/site-packages/coverage/__init__.py
tracer: -none-
CTracer: available
plugins.file_tracers: -none-
plugins.configurers: -none-
plugins.context_switchers: -none-
configs_attempted: .coveragerc
setup.cfg
tox.ini
pyproject.toml
configs_read: -none-
config_file: None
config_contents: -none-
data_file: -none-
python: 3.10.8 (main, Oct 15 2022, 17:48:51) [GCC 9.4.0]
platform: Linux-5.4.0-135-generic-x86_64-with-glibc2.31
implementation: CPython
executable: /home/altendky/tmp/e/venv/bin/python3
def_encoding: utf-8
fs_encoding: utf-8
pid: 60786
cwd: /home/altendky/tmp/e
path: /home/altendky/tmp/e/venv/bin
/home/altendky/.pyenv/versions/3.10.8/lib/python310.zip
/home/altendky/.pyenv/versions/3.10.8/lib/python3.10
/home/altendky/.pyenv/versions/3.10.8/lib/python3.10/lib-dynload
/home/altendky/tmp/e/venv/lib/python3.10/site-packages
environment: HOME = /home/altendky
PYENV_ROOT = /home/altendky/.pyenv
PYENV_SHELL = fish
PYTHONDONTWRITEBYTECODE = 1
PYTHON_CONFIGURE_OPTS = --enable-shared
command_line: venv/bin/coverage debug sys
sqlite3_sqlite_version: 3.31.1
sqlite3_temp_store: 0
sqlite3_compile_options: COMPILER=gcc-9.4.0, ENABLE_COLUMN_METADATA, ENABLE_DBSTAT_VTAB,
ENABLE_FTS3, ENABLE_FTS3_PARENTHESIS, ENABLE_FTS3_TOKENIZER, ENABLE_FTS4,
ENABLE_FTS5, ENABLE_JSON1, ENABLE_LOAD_EXTENSION, ENABLE_PREUPDATE_HOOK,
ENABLE_RTREE, ENABLE_SESSION, ENABLE_STMTVTAB, ENABLE_UNLOCK_NOTIFY,
ENABLE_UPDATE_DELETE_LIMIT, HAVE_ISNAN, LIKE_DOESNT_MATCH_BLOBS,
MAX_SCHEMA_RETRY=25, MAX_VARIABLE_NUMBER=250000, OMIT_LOOKASIDE,
SECURE_DELETE, SOUNDEX, TEMP_STORE=1, THREADSAFE=1, USE_URI
- What versions of what packages do you have installed? The output of
pip freezeis helpful.
$ venv/bin/pip freeze --all
coverage @ git+https://github.com/nedbat/coveragepy@d20c1d0329b585920cf8b1a36af5d2d9c4bba155
pip==22.2.2
setuptools==63.2.0
- What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix.
- https://github.com/altendky/example001/tree/8a76e61a32c77a1ce00699d06dc79d00e4ae9f7d
- What commands did you run?
I deleted the venv and executed ./go.sh &> output.
https://github.com/altendky/example001/blob/8a76e61a32c77a1ce00699d06dc79d00e4ae9f7d/go.sh
#!/bin/bash
set -vx
python3 -m venv venv
venv/bin/python --version --version
#venv/bin/pip install coverage==6.5.0
venv/bin/pip install 'coverage @ git+https://github.com/nedbat/coveragepy@d20c1d0329b585920cf8b1a36af5d2d9c4bba155'
venv/bin/coverage debug sys
venv/bin/pip freeze --all
for label in no_source source source_pkgs
do
venv/bin/coverage erase
venv/bin/coverage run --rcfile coveragerc.${label} -m tests
venv/bin/coverage xml --rcfile coveragerc.${label} -o coverage.${label}.xml
cat coverage.${label}.xml
done
Expected behavior
- I expect that filtering the code to be measured using
[run]source=will not cause loss of coverage data. - I expect that
[run]source=andsource_pkgs=will behave the same except that the latter will not match filesystem paths, only Python packages.
Additional context The real case of this occurred, including the symptoms I first noticed in incorrect diff-cover output, in https://github.com/Chia-Network/chia-blockchain/pull/14141 at https://github.com/Chia-Network/chia-blockchain/pull/14141/commits/b1e479b55ee2f4565bf4901b661fe3a8af76533c and before.