Scopes of several files are merged when specifying a config using `-c`

[fjarri]

Setup:

Python 3.9.9, pip list:

attrs      21.4.0
iniconfig  1.1.1
packaging  21.3
pip        21.3.1
pluggy     1.0.0
py         1.11.0
pyparsing  3.0.7
pytest     7.0.1
setuptools 60.5.0
tomli      2.0.1
wheel      0.37.1

(basically just whatever pip install pytest installs)

Files:

# <dir>/test/test_file1.py

import pytest

@pytest.fixture(autouse=True)
def some_fixture():
    print("Fixture called")

def test_in_file1():
    print("test_in_file1")

# <dir>/test/test_file2.py

def test_in_file2():
    print("test_in_file2")

# <dir>/config/pytest.ini
[pytest]

Running with specifying the config from config explicitly executes the fixture for both tests: py.test -c config/pytest.ini -s -v tests/test_file1.py tests/test_file2.py

config/::test_in_file1 Fixture called
test_in_file1
PASSED
config/::test_in_file2 Fixture called
test_in_file2
PASSED

Omitting the config, using a config from the root directory <dir>, or not specifying the test files explicitly (just py.test -c config/pytest.ini -s -v):

config/tests/test_file1.py::test_in_file1 Fixture called
test_in_file1
PASSED
config/tests/test_file2.py::test_in_file2 test_in_file2
PASSED

which is the behavior I would expect.

Downgrading to pytest==6.2.5 and running the command above (py.test -c config/pytest.ini -s -v tests/test_file1.py tests/test_file2.py) does not lead to the bug either. Interestingly enough, the "config" prefix is missing from the output:

tests/test_file1.py::test_in_file1 Fixture called
test_in_file1
PASSED
tests/test_file2.py::test_in_file2 test_in_file2
PASSED

Must be some config-based grouping feature introduced in 7?

[RonnyPfannschmidt]

at first glance this is a issue with node id computation messing up with relative pathnames, thanks for providing the comprehensible example, we need to add it to the testsuite

[symonk]

I think the problem here is both node ids are:

::test_in_file1
::test_in_file2

when ::test_in_file1 resolves its autouse fixtures it populates _nodeid_autousenames with an empty string key " so the parentnodeid for ::test_in_file2 looks it up in the dictionary and returns any autouse fixtures defined previously by ::test_in_file1 - I'm not sure why those nodeids are like that; but i'll take a look after lunch. I'd expect:

test/test_file1.py::test_in_file1
test/test_file2.py::test_in_file2

as nodeids respectfully here, heres a test that passes due to the breakage if anyone beats me to it!

def test_passing_config_works_correctly(pytester: pytest.Pytester) -> None:
    """Regression for #9703"""
    test_sub = pytester.mkdir("test")
    test_sub.joinpath("test_file1.py").write_text(
        textwrap.dedent(
            """
            import pytest

            @pytest.fixture(autouse=True)
            def some_fixture():
                print("Fixture called")

            def test_in_file1():
                print("test_in_file1")
            """
        )
    )
    test_sub.joinpath("test_file2.py").write_text(
        textwrap.dedent(
            """
            import pytest
            def test_in_file2():
                print("test_in_file2")
            """
        )
    )
    config_sub = pytester.mkdir("config")
    config_sub.joinpath("pytest.ini").write_text(
        textwrap.dedent(
            """
            [pytest]
            """
        )
    )
    output = pytester.runpytest(
        "-c",
        "config/pytest.ini",
        "-s",
        "-v",
        "test/test_file1.py",
        "test/test_file2.py",
    )
    output.stdout.fnmatch_lines("test/test_file1.py::test_in_file1 Fixture called")
    output.stdout.fnmatch_lines("test/test_file2.py::test_in_file2 test_in_file2")

Think this largely stems from rootdir being set to the config directory and the paths provided are technically not 'relative' to it; unfortunately didn't get much time today to look at it due to something unexpected that came up.

essentially the same as running pytest --rootdir config/ -s -v test/test_file1.py test/test_file2.py - problem is; I'm not sure on the appropriate 'fix' for this; is it partially human error here and we need to provide better errors? not quite sure yet

[fjarri]

essentially the same as running pytest --rootdir config/ -s -v test/test_file1.py test/test_file2.py

Why does specifying a config file set the rootdir? Shouldn't it be independent?

[symonk]

I don't know enough about the internals to answer that but I'm happy to look further into it when I wrap up some other things, hopefully someone in the know can give us some reasoning on that