tests/test_pytest_cov.py::test_subprocess_with_path_aliasing fails if coverage is installed outside venv

[mgorny]

Summary

For the purpose of Gentoo packaging, I'm trying to run pytest-cov tests inside a venv with --system-site-packages enabled. However, for some reason tests/test_pytest_cov.py::test_subprocess_with_path_aliasing is failing if coverage is installed outside the venv. I'd really appreciate some help with figuring out why this is happening.

Expected vs actual result

_________________________________________________ test_subprocess_with_path_aliasing __________________________________________________

testdir = <Testdir local('/tmp/pytest-of-mgorny/pytest-26/test_subprocess_with_path_aliasing0')>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f916f346410>

    def test_subprocess_with_path_aliasing(testdir, monkeypatch):
        src = testdir.mkdir('src')
        src.join('parent_script.py').write(SCRIPT_PARENT)
        src.join('child_script.py').write(SCRIPT_CHILD)
        aliased = testdir.mkdir('aliased')
        parent_script = aliased.join('parent_script.py')
        parent_script.write(SCRIPT_PARENT)
        aliased.join('child_script.py').write(SCRIPT_CHILD)
    
        testdir.tmpdir.join('.coveragerc').write("""
    [paths]
    source =
        src
        aliased
    [run]
    source =
        parent_script
        child_script
    parallel = true
    """)
    
        monkeypatch.setitem(os.environ, 'PYTHONPATH', os.pathsep.join([
            os.environ.get('PYTHONPATH', ''), 'aliased']))
        result = testdir.runpytest('-v',
                                   '--cov',
                                   '--cov-report=term-missing',
                                   parent_script)
    
>       result.stdout.fnmatch_lines([
            '*- coverage: platform *, python * -*',
            'src[\\/]child_script* %s*' % CHILD_SCRIPT_RESULT,
            'src[\\/]parent_script* %s*' % PARENT_SCRIPT_RESULT,
        ])
E       Failed: nomatch: '*- coverage: platform *, python * -*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /tmp/pytest-cov/.venv/bin/python'
E           and: 'cachedir: .pytest_cache'
E           and: 'rootdir: /tmp/pytest-of-mgorny/pytest-26/test_subprocess_with_path_aliasing0'
E           and: 'plugins: cov-3.0.0, pkgcore-0.12.9, xprocess-0.18.1, freezegun-0.4.2, forked-1.4.0, xdist-2.5.0, anyio-3.5.0'
E           and: 'collecting ... collected 2 items'
E           and: ''
E           and: 'aliased/parent_script.py::test_foo[0] PASSED                             [ 50%]'
E           and: 'aliased/parent_script.py::test_foo[1] PASSED                             [100%]'
E           and: ''
E       fnmatch: '*- coverage: platform *, python * -*'
E          with: '---------- coverage: platform linux, python 3.10.2-final-0 -----------'
E       nomatch: 'src[\\/]child_script* [56] * 100%*'
E           and: 'Name                   Stmts   Miss  Cover   Missing'
E           and: '----------------------------------------------------'
E           and: 'src/parent_script.py       9      0   100%'
E           and: '----------------------------------------------------'
E           and: 'TOTAL                      9      0   100%'
E           and: ''
E           and: ''
E           and: '============================== 2 passed in 0.19s ==============================='
E           and: "pytest-xprocess reminder::Be sure to terminate the started process by running 'pytest --xkill' if you have not explicitly done so in your fixture with 'xprocess.getinfo(<process_name>).terminate()'."
E       remains unmatched: 'src[\\/]child_script* [56] * 100%*'

/tmp/pytest-cov/tests/test_pytest_cov.py:605: Failed
-------------------------------------------------------- Captured stdout call ---------------------------------------------------------
running: /tmp/pytest-cov/.venv/bin/python -mpytest --basetemp=/tmp/pytest-of-mgorny/pytest-26/test_subprocess_with_path_aliasing0/runpytest-0 -v --cov --cov-report=term-missing /tmp/pytest-of-mgorny/pytest-26/test_subprocess_with_path_aliasing0/aliased/parent_script.py --basetemp=/tmp/pytest-of-mgorny/pytest-26/basetemp
     in: /tmp/pytest-of-mgorny/pytest-26/test_subprocess_with_path_aliasing0
============================= test session starts ==============================
platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /tmp/pytest-cov/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/pytest-of-mgorny/pytest-26/test_subprocess_with_path_aliasing0
plugins: cov-3.0.0, pkgcore-0.12.9, xprocess-0.18.1, freezegun-0.4.2, forked-1.4.0, xdist-2.5.0, anyio-3.5.0
collecting ... collected 2 items

aliased/parent_script.py::test_foo[0] PASSED                             [ 50%]
aliased/parent_script.py::test_foo[1] PASSED                             [100%]

---------- coverage: platform linux, python 3.10.2-final-0 -----------
Name                   Stmts   Miss  Cover   Missing
----------------------------------------------------
src/parent_script.py       9      0   100%
----------------------------------------------------
TOTAL                      9      0   100%


============================== 2 passed in 0.19s ===============================

Reproducer

First, install all necessary test deps (most notably coverage) to the system site-packages directory. Then, inside pytest-cov sources:

python -m venv --system-site-packages .venv
. .venv/bin/activate
pip install .
python -m pytest -vv tests/test_pytest_cov.py::test_subprocess_with_path_aliasing

Curious enough, if I copy coverage inside the venv, the test suddenly starts working:

cp -r /usr/lib/python3.10/site-packages/coverage .venv/lib/python3.10/site-packages/
python -m pytest -vv tests/test_pytest_cov.py::test_subprocess_with_path_aliasing

Versions

pip list: pip-list.txt

$ python --version
Python 3.10.2
$ python -m pytest --version
pytest 6.2.5
$ git rev-parse HEAD
f00ccf94791c8b3e777c12f34b2caa9689c3047a

[ionelmc]

It fails because the special pth file isn't loaded. Maybe just skip that test if you can't install it properly for the purpose of testing?

[mgorny]

But why wouldn't it be loaded? And why does symlinking coverage package change the result?

[ionelmc]

Probably this part fails in some weird way, in the sense that certain assumptions are broken (eg: system site packages are setup with something that breaks when PYTHONPATH is overriden?):

monkeypatch.setitem(os.environ, 'PYTHONPATH', os.pathsep.join([
            os.environ.get('PYTHONPATH', ''), 'aliased']))