coveragepy icon indicating copy to clipboard operation
coveragepy copied to clipboard

Published 20 hours ago • verified publisher icon nedbat

Coverage.py fails to omit *third-party* coverage in a nested venv from the same implicit namespace package

Open webknjaz opened this issue 4 months ago • 2 comments

Describe the bug

I have a case when I get unexpected coverage measurements from a tox-managed virtualenv existing within the source tree. Coveragepy is invoked via pytest-cov but all configuration is in coveragerc so I'm pretty sure this issue belongs here.

This reminds me of #876 and #905. Having stared at https://github.com/nedbat/coveragepy/commit/0285af966a3942d8bd63489bd285328e96221126 and https://github.com/nedbat/coveragepy/commit/5c2f614e01d35271f7907d85050115071cf24e87, I concluded that the fixes are still imperfect.

The reproducers in those issues showcased regular non-namespaces packages, which is why it wasn't on anybody's radar, I think. But I'm sure my case is legit too.

In the example below, awx_plugins is the namespace. The dependency is the awx_plugins.interfaces importable and the project itself contains awx_plugins.credentials and awx_plugins.inventory. The project's files are under src/awx_plugins/{credentials,inventory}/*.py and tests/*.py. The dependecy has src/awx_plugins/interfaces/*.py in the repo, but all are installed side-by-side as .tox/py/lib/python3.13/site-packages/awx_plugins/{credentials,interfaces,inventory}/*.py. .tox/py/lib/python3.13/site-packages/awx_plugins/__init__.py does not exist. .tox/py/lib/python3.13/site-packages/awx_plugins/{credentials,interfaces,inventory}/__init__.py do not exist either.

The project is configured with

[paths]
_site-packages-to-src-mapping =
  src
  */src
  *\src
  */lib/pypy*/site-packages
  */lib/python*/site-packages
  *\Lib\site-packages

which maps all the project-local source files properly (which is visible in the report correctly). The third-party ones remain unmatched and show up with site-packages paths.

There's also

[run]
relative_files = true
source =
  .
source_pkgs =
  awx_plugins

The generic paths and namespaces effectively give me the ability to copy the good coveragerc file over into similar packages of the same namespace and folder layout. This is intentional.

I'm almost sure changing source_pkgs to list awx_plugins.credentials and awx_plugins.inventory would limit what ends up in the report but that's not something I want given that I wish to have a reusable config across an ecosystem of similar projects (i.e. plugins to something else).

I've tested that adding

omit =
  .tox/**

is a functional workaround, following the suggestions in other issues.

But ultimately, I believe that this is a bug fixable within coveragepy and I shouldn't have to keep the omit settings in my configs.

To Reproduce

  1. What version of Python are you using? A range of Python 3.11-3.13, I'm pretty sure the problem is agnostic, though.
  2. What version of coverage.py shows the problem?
    👉 click to unfold 👈 
    $ .tox/py/bin/python -m coverage debug sys
-- sys -------------------------------------------------------
               coverage_version: 7.6.1
                coverage_module: ~/src/github/ansible/awx-plugins/.tox/py/lib/python3.13/site-packages/coverage/__init__.py
                           core: -none-
                        CTracer: available
           plugins.file_tracers: -none-
            plugins.configurers: covdefaults.CovDefaults
      plugins.context_switchers: -none-
              configs_attempted: ~/src/github/ansible/awx-plugins/.coveragerc
                   configs_read: ~/src/github/ansible/awx-plugins/.coveragerc
                    config_file: ~/src/github/ansible/awx-plugins/.coveragerc
                config_contents: b'[html]\nshow_contexts = true\nskip_covered = false\n\n[paths]\n_site-packages-to-src-mapping =\n  src\n  */src\n  *\\src\n  */lib/pypy*/site-packages\n  */lib/python*/site-packages\n  *\\Lib\\site-packages\n\n[report]\n# `fail_under` is set here temporarily until it can be dropped:\nfail_under = 39.27\nskip_covered = true\nskip_empty = true\nshow_missing = true\nexclude_also =\n  ^\\s*@pytest\\.mark\\.xfail\n\n[run]\nbranch = true\ncover_pylib = false\n# https://coverage.rtfd.io/en/latest/contexts.html#dynamic-contexts\n# dynamic_context = test_function  # conflicts with `pytest-cov` if set here\nomit =\n  .tox/**\nparallel = true\nplugins =\n  covdefaults\nrelative_files = true\nsource =\n  .\nsource_pkgs =\n  awx_plugins\n'
                      data_file: -none-
                       python: 3.13.0b2 (main, Jun 23 2024, 13:25:09) [GCC 13.2.1 20240113]
                       platform: Linux-6.6.13-gentoo-dist-x86_64-Intel-R-_Core-TM-_i7-9850H_CPU_@_2.60GHz-with-glibc2.38
                 implementation: CPython
                    gil_enabled: True
                     executable: ~/src/github/ansible/awx-plugins/.tox/py/bin/python
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 31740
                            cwd: ~/src/github/ansible/awx-plugins
                           path: ~/src/github/ansible/awx-plugins
                                 ~/.pyenv/versions/3.13.0b2/lib/python313.zip
                                 ~/.pyenv/versions/3.13.0b2/lib/python3.13
                                 ~/.pyenv/versions/3.13.0b2/lib/python3.13/lib-dynload
                                 ~/src/github/ansible/awx-plugins/.tox/py/lib/python3.13/site-packages
                                 ~/src/github/ansible/awx-plugins/src
                    environment: HOME = ~
                                 PYENV_ROOT = ~/.pyenv
                                 PYENV_SHELL = zsh
                                 PYENV_VIRTUALENV_INIT = 1
                   command_line: ~/src/github/ansible/awx-plugins/.tox/py/lib/python3.13/site-packages/coverage/__main__.py debug sys
         sqlite3_sqlite_version: 3.44.2
             sqlite3_temp_store: 0
        sqlite3_compile_options: ATOMIC_INTRINSICS=1, COMPILER=gcc-13.2.1 20240113, DEFAULT_AUTOVACUUM,
                                 DEFAULT_CACHE_SIZE=-2000, DEFAULT_FILE_FORMAT=4,
                                 DEFAULT_JOURNAL_SIZE_LIMIT=-1, DEFAULT_MMAP_SIZE=0, DEFAULT_PAGE_SIZE=4096,
                                 DEFAULT_PCACHE_INITSZ=20, DEFAULT_RECURSIVE_TRIGGERS,
                                 DEFAULT_SECTOR_SIZE=4096, DEFAULT_SYNCHRONOUS=2,
                                 DEFAULT_WAL_AUTOCHECKPOINT=1000, DEFAULT_WAL_SYNCHRONOUS=2,
                                 DEFAULT_WORKER_THREADS=0, ENABLE_API_ARMOR, ENABLE_BYTECODE_VTAB,
                                 ENABLE_COLUMN_METADATA, ENABLE_DBPAGE_VTAB, ENABLE_DBSTAT_VTAB,
                                 ENABLE_EXPLAIN_COMMENTS, ENABLE_FTS3, ENABLE_FTS3_PARENTHESIS, ENABLE_FTS4,
                                 ENABLE_FTS5, ENABLE_GEOPOLY, ENABLE_HIDDEN_COLUMNS, ENABLE_ICU,
                                 ENABLE_MATH_FUNCTIONS, ENABLE_MEMSYS5, ENABLE_NORMALIZE,
                                 ENABLE_OFFSET_SQL_FUNC, ENABLE_PREUPDATE_HOOK, ENABLE_RBU, ENABLE_RTREE,
                                 ENABLE_SESSION, ENABLE_STMTVTAB, ENABLE_STMT_SCANSTATUS,
                                 ENABLE_UNKNOWN_SQL_FUNCTION, ENABLE_UNLOCK_NOTIFY,
                                 ENABLE_UPDATE_DELETE_LIMIT, HAVE_ISNAN, MALLOC_SOFT_LIMIT=1024,
                                 MAX_ATTACHED=10, MAX_COLUMN=2000, MAX_COMPOUND_SELECT=500,
                                 MAX_DEFAULT_PAGE_SIZE=8192, MAX_EXPR_DEPTH=1000, MAX_FUNCTION_ARG=127,
                                 MAX_LENGTH=1000000000, MAX_LIKE_PATTERN_LENGTH=50000,
                                 MAX_MMAP_SIZE=0x7fff0000, MAX_PAGE_COUNT=1073741823, MAX_PAGE_SIZE=65536,
                                 MAX_SQL_LENGTH=1000000000, MAX_TRIGGER_DEPTH=1000,
                                 MAX_VARIABLE_NUMBER=32766, MAX_VDBE_OP=250000000, MAX_WORKER_THREADS=8,
                                 MUTEX_PTHREADS, SOUNDEX, SYSTEM_MALLOC, TEMP_STORE=1, THREADSAFE=1, USE_URI
  3. What versions of what packages do you have installed?
    👉 click to unfold 👈 
    $ .tox/py/bin/python -m pip freeze        
adal==1.2.7
attrs==24.2.0
awx-plugins-core @ file://~/src/github/ansible/awx-plugins/.tox/.tmp/package/97/awx_plugins_core-0.0.1a5.dev213%2Bg8ea07661f9.d20241004-0.editable-py3-none-any.whl#sha256=3ca7389cb589da068511245e32662a41eefebf9ecfbbdfa7c41025be26f55c12
awx_plugins.interfaces @ git+https://github.com/ansible/awx_plugins.interfaces.git@3a78f59ffe922638f37bf63b4243aac41ed8abd8
azure-core==1.31.0
azure-identity==1.18.0
azure-keyvault==4.2.0
azure-keyvault-certificates==4.8.0
azure-keyvault-keys==4.9.0
azure-keyvault-secrets==4.8.0
boto3==1.35.31
botocore==1.35.31
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.3.2
covdefaults==2.3.0
coverage==7.6.1
coverage-enable-subprocess==1.0
cryptography==43.0.1
execnet==2.1.1
hypothesis==6.112.2
idna==3.10
iniconfig==2.0.0
isodate==0.6.1
jmespath==1.0.1
msal==1.31.0
msal-extensions==1.2.0
msrest==0.7.1
msrestazure==0.6.4.post1
oauthlib==3.2.2
packaging==24.1
pluggy==1.5.0
portalocker==2.10.1
pycparser==2.22
PyJWT==2.9.0
pytest==8.3.3
pytest-cov==5.0.0
pytest-mock==3.14.0
pytest-xdist==3.6.1
python-dateutil==2.9.0.post0
python-dsv-sdk==1.0.4
python-tss-sdk==1.2.3
PyYAML==6.0.2
requests==2.32.3
requests-oauthlib==2.0.0
s3transfer==0.10.2
six==1.16.0
sortedcontainers==2.4.0
typing_extensions==4.12.2
urllib3==2.2.3
  4. What code shows the problem? https://github.com/ansible/awx-plugins/commit/8ea07661f9da09dedaa52b569978c448fb871931
  5. What commands should we run to reproduce the problem? 
    $ python3 -Im pip install tox
$ git clone https://github.com/ansible/awx-plugins.git
$ cd awx-plugins
$ python3 -Im tox
$ python3 -Im webbrowser .tox/py/tmp/htmlcov/index.html

That shows a couple of files under .tox/py/lib/python3.13/site-packages/awx_plugins/interfaces/**. They should not be listed.

Expected behavior

Coveragepy should not collect coverage in third-party modules from the same implicit namespace, installed in the same virtualenv.

Additional context

The concrete example is inspectable at https://app.codecov.io/gh/ansible/awx-plugins/commit/8ea07661f9da09dedaa52b569978c448fb871931/tree?flags%5B0%5D=pytest. The pre-selected pytest flag is what you want to have selected at all times since there's also MyPy coverage that's overlayed if you deselect it.

webknjaz avatar Oct 04 '24 14:10 webknjaz