Test class instance is not recreated when retrying tests

[ngg]

Since upgrading to pytest>=7.0.0, the self parameter between test runs remain the same when retrying a single test function within a test class. When using pytest==6.5.2, it is working as expected, the self parameter refers to a new instance every time the test is run.

To reproduce the issue, you can use the following code:

import pytest
from flaky import flaky

class TestExample:
    @property
    def counter(self):
        self.__dict__.setdefault('_counter', 0)
        self._counter += 1
        return self._counter

    @pytest.fixture(autouse=True)
    def some_fixture(self):
        print('SETUP', self.counter)
        yield
        print('TEARDOWN', self.counter)

    @flaky(max_runs=5)
    def test_something(self, param=[]):
        print('TEST', self.counter)
        param.append(0)
        assert len(param) > 2

    def test_other(self):
        print('OTHER', self.counter)

Running these tests output the following, showing that the same instance is used for self when retrying the test:

SETUP 1
TEST 2
TEARDOWN 3
SETUP 4
TEST 5
TEARDOWN 6
SETUP 7
TEST 8
.TEARDOWN 9
SETUP 1
OTHER 2
.TEARDOWN 3

I think this should be considered as a bug, see for example the commit message of https://github.com/pytest-dev/pytest/commit/0dc036035107b213c9b73bf965cbd7356111b85a which states This is definitely broken since it breaks test isolation. for a similar case.

I could work around this issue in a really ugly and fragile way using a pytest hook like this, but I think this should be fixed properly within flaky:

import pytest
import weakref
from typing import Any, Generator

last_weak_obj_key = pytest.StashKey[weakref.ref[Any]]()
last_weak_instance_key = pytest.StashKey[weakref.ref[Any]]()


@pytest.hookimpl(hookwrapper=True, tryfirst=True)
def pytest_runtest_setup(item: Any) -> Generator[None, Any, None]:
    last_weak_obj = item.stash.get(last_weak_obj_key, lambda: None)
    last_weak_instance = item.stash.get(last_weak_instance_key, lambda: None)

    assert item.obj is not None
    if item.obj is last_weak_obj() or (
        item.instance is not None and item.instance is last_weak_instance()
    ):
        # Deleting these two internal attributes trigger new instance creation when accessing item.obj or item.instance the next time
        del item._obj
        del item._instance

    assert item.obj is not None and item.obj is not last_weak_obj()
    item.stash[last_weak_obj_key] = weakref.ref(item.obj)

    assert item.instance is None or item.instance is not last_weak_instance()
    item.stash[last_weak_instance_key] = weakref.ref(item.instance)

    yield

As pytest-rerunfailures is also affected, I've opened this issue there as well: https://github.com/pytest-dev/pytest-rerunfailures/issues/268