improve(fixtures-per-test): exclude pseudo fixtures from output #11295

[WarrenTheRabbit]

Closes #11295 by excluding 'pseudo fixtures' from the --fixtures-per-test output.

pseudo fixtures are neither builtin fixtures nor ones created by the user with the pytest.fixture decorator. They are created internally by pytest when @pytest.mark.parametrize directly parametrizes a test. It provides a succinct syntax for batch executing a test over multiple argument sets.

For example, when pytest --fixtures-per-test is run, this test

# test_file.py
Import pytest

@pytest.mark.parametrize("x", [1, 2])
def test_one(monkeypatch, x):
    pass

will no longer produce output such as this:

------------------------- fixtures used by test_one[1] -------------------------
----------------------------------- (test_file.py:5) ------------------------------
monkeypatch -- src/_pytest/monkeypatch.py:33
    A convenient fixture for monkey-patching.
x -- src/_pytest/python.py:1113
    no docstring available
    
------------------------- fixtures used by test_one[2] -------------------------
----------------------------------- (test_file.py:5) ------------------------------
monkeypatch -- src/_pytest/monkeypatch.py:33
    A convenient fixture for monkey-patching.
x -- src/_pytest/python.py:1113
    no docstring available

Instead, it will be:

------------------------- fixtures used by test_one[1] -------------------------
----------------------------------- (test_file.py:5) ------------------------------
monkeypatch -- src/_pytest/monkeypatch.py:33
    A convenient fixture for monkey-patching.
    
------------------------- fixtures used by test_one[2] -------------------------
----------------------------------- (test_file.py:5) ------------------------------
monkeypatch -- src/_pytest/monkeypatch.py:33
    A convenient fixture for monkey-patching.

Justification for changing the output

The original output did not match with new users' intuitions and expectations

As a new user, I found it unintuitive to see the @pytest.mark.parametrize variables appear in my --fixtures-per-test report. I am of the opinion that the inclusion of pseudo fixtures in the output confuses new users because they do not conform to the expectations established in the documentation. Namely, that fixtures are

• richly reusable
• provide setup/teardown features
• created via the @pytest.fixture decorator

The original output puts attention on internal implementation details

The purpose of --fixtures-per-test is to create a summary of the user's fixture decisions and dependencies. Yet creating a fixture for the user is not the goal of the direct parametrization mark. Instead, _pytest'_s internals just leverage the fixture system to achieve the actual goal: a succinct batch execution syntax. I believe that including the pseudo fixtures in the output exposes internal implementation details unnecessarily and distracts from the user-side summary.

Checklist

• [X] Include new tests or update existing tests when applicable.
• [ ] I have not found any documentation to update yet.
• [ ] Allow maintainers to push and squash when merging my commits. Please uncheck this if you prefer to squash the commits yourself.
• [X] Add text like closes #XYZW to the PR . . .
• [X] Create a new changelog file . . .
• [X] Add yourself to AUTHORS in alphabetical order.

[WarrenTheRabbit]

Issue #12086 suggests another possible benefit of removing pseudo fixtures from the --fixtures-per-test output. If a user is expecting their fixture, user_defined, to be parametrised in the following manner:

@pytest.fixture
def user_defined(request):
   return request.param * 3
 
@pytest.mark.parametrize('user_defined', [1])
def test_function(user_defined):
    assert user_defined == 3

This test will fail:

user_defined = 1

    @pytest.mark.parametrize('user_defined', [1])
    def test_scenario_1(user_defined):
>       assert user_defined == 3
E       assert 1 == 3

file.py:9: AssertionError

Yet the current output of --fixtures-per-test does not provide a clue about what is happening. Even though the user-defined fixture has been shadowed by parametrization and is no longer in the test function's scope, it doesn't look like that has happened:

--------------------------- fixtures used by test_scenario_1[1] ----------------------------
--------------------------------------- (file.py:8) ----------------------------------------
user_defined -- src/_pytest/python.py:1113
    no docstring available

But if pseudo fixtures are excluded from the output, there is a stronger message about the user-defined fixtures's non-use because no fixture use will be reported.

Documentation needs to clarify

From one point of view, one mystery has been replaced with another, so I definitely think documentation needs to be available to help clarify the matter for new users.

Devil's advocate

Should --fixtures-per-test still include pseudo fixtures but add a docstring that explains what they are? In my opinion, this is very explicit, which is good, and it will provide some comfort to users when they use --fixtures-per-test and have their intuitions about fixtures in pytest violated, but I suppose it still has the problem of exposing internal details.