coveragepy icon indicating copy to clipboard operation
coveragepy copied to clipboard

Reports using multiple `[run]` `source=` entries can lose data from colliding top level files

Open altendky opened this issue 2 years ago • 0 comments

Describe the bug

  1. When using [run] source= with multiple values at least top level files can collide resulting in missing data in a report such as coverage.xml. Specifically in my case, my package/__init__.py and tests/__init__.py collide. This behavior feels more like what I would expect from [paths] source=.
  2. Comparing [run] source= to source_pkgs= with multiple values shows more differences than I would expect from the docs. This fixes 1), only a single <source> entry is present which is empty (was not empty with 6.5.0) in the coverage.xml. With 6.5.0, that single entry was an absolute path instead of a relative path.

If these two things both seem like issue but my judgement that they are the same thing is wrong, certainly we can relocate 2) to a separate issue. Also, the side note of the empty <source></source> tag in the XML output might be another issue, I only just now noticed it when running against master to create this report.

Note in the reports below how coverage.source.xml only has results for the tests/__init__.py file and not for package/__init__.py.

https://github.com/altendky/example001/blob/8a76e61a32c77a1ce00699d06dc79d00e4ae9f7d/coverage.no_source.xml

Click to see `coverage.no_source.xml`
<?xml version="1.0" ?>
<coverage version="7.0.0b2.dev1" timestamp="1671205057420" lines-valid="9" lines-covered="9" line-rate="1" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
	<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.0.0b2.dev1 -->
	<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
	<sources>
		<source></source>
	</sources>
	<packages>
		<package name="package" line-rate="1" branch-rate="0" complexity="0">
			<classes>
				<class name="__init__.py" filename="package/__init__.py" complexity="0" line-rate="1" branch-rate="0">
					<methods/>
					<lines>
						<line number="1" hits="1"/>
						<line number="2" hits="1"/>
						<line number="4" hits="1"/>
						<line number="5" hits="1"/>
					</lines>
				</class>
			</classes>
		</package>
		<package name="tests" line-rate="1" branch-rate="0" complexity="0">
			<classes>
				<class name="__init__.py" filename="tests/__init__.py" complexity="0" line-rate="1" branch-rate="0">
					<methods/>
					<lines>
						<line number="1" hits="1"/>
						<line number="3" hits="1"/>
						<line number="4" hits="1"/>
						<line number="6" hits="1"/>
						<line number="7" hits="1"/>
					</lines>
				</class>
				<class name="__main__.py" filename="tests/__main__.py" complexity="0" line-rate="1" branch-rate="0">
					<methods/>
					<lines/>
				</class>
			</classes>
		</package>
	</packages>
</coverage>

https://github.com/altendky/example001/blob/8a76e61a32c77a1ce00699d06dc79d00e4ae9f7d/coverage.source.xml

Click to see `coverage.source.xml`
<?xml version="1.0" ?>
<coverage version="7.0.0b2.dev1" timestamp="1671205057646" lines-valid="9" lines-covered="9" line-rate="1" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
	<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.0.0b2.dev1 -->
	<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
	<sources>
		<source>package</source>
		<source>tests</source>
	</sources>
	<packages>
		<package name="." line-rate="1" branch-rate="0" complexity="0">
			<classes>
				<class name="__init__.py" filename="__init__.py" complexity="0" line-rate="1" branch-rate="0">
					<methods/>
					<lines>
						<line number="1" hits="1"/>
						<line number="3" hits="1"/>
						<line number="4" hits="1"/>
						<line number="6" hits="1"/>
						<line number="7" hits="1"/>
					</lines>
				</class>
				<class name="__main__.py" filename="__main__.py" complexity="0" line-rate="1" branch-rate="0">
					<methods/>
					<lines/>
				</class>
			</classes>
		</package>
	</packages>
</coverage>

https://github.com/altendky/example001/blob/8a76e61a32c77a1ce00699d06dc79d00e4ae9f7d/coverage.source_pkgs.xml

Click to see `coverage.source_pkgs.xml`
<?xml version="1.0" ?>
<coverage version="7.0.0b2.dev1" timestamp="1671205057874" lines-valid="9" lines-covered="9" line-rate="1" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
	<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.0.0b2.dev1 -->
	<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
	<sources>
		<source></source>
	</sources>
	<packages>
		<package name="package" line-rate="1" branch-rate="0" complexity="0">
			<classes>
				<class name="__init__.py" filename="package/__init__.py" complexity="0" line-rate="1" branch-rate="0">
					<methods/>
					<lines>
						<line number="1" hits="1"/>
						<line number="2" hits="1"/>
						<line number="4" hits="1"/>
						<line number="5" hits="1"/>
					</lines>
				</class>
			</classes>
		</package>
		<package name="tests" line-rate="1" branch-rate="0" complexity="0">
			<classes>
				<class name="__init__.py" filename="tests/__init__.py" complexity="0" line-rate="1" branch-rate="0">
					<methods/>
					<lines>
						<line number="1" hits="1"/>
						<line number="3" hits="1"/>
						<line number="4" hits="1"/>
						<line number="6" hits="1"/>
						<line number="7" hits="1"/>
					</lines>
				</class>
				<class name="__main__.py" filename="tests/__main__.py" complexity="0" line-rate="1" branch-rate="0">
					<methods/>
					<lines/>
				</class>
			</classes>
		</package>
	</packages>
</coverage>

To Reproduce How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

  1. What version of Python are you using?
  • CPython 3.10 from pyenv in Ubuntu Linux Python 3.10.8 (main, Oct 15 2022, 17:48:51) [GCC 9.4.0]
  1. What version of coverage.py shows the problem? The output of coverage debug sys is helpful.
-- sys -------------------------------------------------------
               coverage_version: 7.0.0b2.dev1
                coverage_module: /home/altendky/tmp/e/venv/lib/python3.10/site-packages/coverage/__init__.py
                         tracer: -none-
                        CTracer: available
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: .coveragerc
                                 setup.cfg
                                 tox.ini
                                 pyproject.toml
                   configs_read: -none-
                    config_file: None
                config_contents: -none-
                      data_file: -none-
                         python: 3.10.8 (main, Oct 15 2022, 17:48:51) [GCC 9.4.0]
                       platform: Linux-5.4.0-135-generic-x86_64-with-glibc2.31
                 implementation: CPython
                     executable: /home/altendky/tmp/e/venv/bin/python3
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 60786
                            cwd: /home/altendky/tmp/e
                           path: /home/altendky/tmp/e/venv/bin
                                 /home/altendky/.pyenv/versions/3.10.8/lib/python310.zip
                                 /home/altendky/.pyenv/versions/3.10.8/lib/python3.10
                                 /home/altendky/.pyenv/versions/3.10.8/lib/python3.10/lib-dynload
                                 /home/altendky/tmp/e/venv/lib/python3.10/site-packages
                    environment: HOME = /home/altendky
                                 PYENV_ROOT = /home/altendky/.pyenv
                                 PYENV_SHELL = fish
                                 PYTHONDONTWRITEBYTECODE = 1
                                 PYTHON_CONFIGURE_OPTS = --enable-shared
                   command_line: venv/bin/coverage debug sys
         sqlite3_sqlite_version: 3.31.1
             sqlite3_temp_store: 0
        sqlite3_compile_options: COMPILER=gcc-9.4.0, ENABLE_COLUMN_METADATA, ENABLE_DBSTAT_VTAB,
                                 ENABLE_FTS3, ENABLE_FTS3_PARENTHESIS, ENABLE_FTS3_TOKENIZER, ENABLE_FTS4,
                                 ENABLE_FTS5, ENABLE_JSON1, ENABLE_LOAD_EXTENSION, ENABLE_PREUPDATE_HOOK,
                                 ENABLE_RTREE, ENABLE_SESSION, ENABLE_STMTVTAB, ENABLE_UNLOCK_NOTIFY,
                                 ENABLE_UPDATE_DELETE_LIMIT, HAVE_ISNAN, LIKE_DOESNT_MATCH_BLOBS,
                                 MAX_SCHEMA_RETRY=25, MAX_VARIABLE_NUMBER=250000, OMIT_LOOKASIDE,
                                 SECURE_DELETE, SOUNDEX, TEMP_STORE=1, THREADSAFE=1, USE_URI
  1. What versions of what packages do you have installed? The output of pip freeze is helpful.
$ venv/bin/pip freeze --all
coverage @ git+https://github.com/nedbat/coveragepy@d20c1d0329b585920cf8b1a36af5d2d9c4bba155
pip==22.2.2
setuptools==63.2.0
  1. What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix.
  • https://github.com/altendky/example001/tree/8a76e61a32c77a1ce00699d06dc79d00e4ae9f7d
  1. What commands did you run?

I deleted the venv and executed ./go.sh &> output.

https://github.com/altendky/example001/blob/8a76e61a32c77a1ce00699d06dc79d00e4ae9f7d/go.sh

#!/bin/bash

set -vx

python3 -m venv venv
venv/bin/python --version --version
#venv/bin/pip install coverage==6.5.0
venv/bin/pip install 'coverage @ git+https://github.com/nedbat/coveragepy@d20c1d0329b585920cf8b1a36af5d2d9c4bba155'

venv/bin/coverage debug sys
venv/bin/pip freeze --all

for label in no_source source source_pkgs
do
  venv/bin/coverage erase
  venv/bin/coverage run --rcfile coveragerc.${label} -m tests
  venv/bin/coverage xml --rcfile coveragerc.${label} -o coverage.${label}.xml
  cat coverage.${label}.xml
done

Expected behavior

  1. I expect that filtering the code to be measured using [run] source= will not cause loss of coverage data.
  2. I expect that [run] source= and source_pkgs= will behave the same except that the latter will not match filesystem paths, only Python packages.

Additional context The real case of this occurred, including the symptoms I first noticed in incorrect diff-cover output, in https://github.com/Chia-Network/chia-blockchain/pull/14141 at https://github.com/Chia-Network/chia-blockchain/pull/14141/commits/b1e479b55ee2f4565bf4901b661fe3a8af76533c and before.

altendky avatar Dec 16 '22 15:12 altendky