[v7.0.0 regression?] AssertionError: previous item was not torn down properly

[webknjaz]

I'm trying to update from pytest v6.2.5 to v7.0.0 via https://github.com/cherrypy/cheroot/pull/475 and it fails the testing weirdly with $sbj.

The assertion error in question comes from the pytest's internals. Looking at the traceback promptly suggests that it's happening when one test module is being set up but the previous module hasn't been torn down completely: https://github.com/pytest-dev/pytest/blob/f28421cc7068b13ba63c1f60cc21f898cccea36c/src/_pytest/runner.py#L489.

I don't know how it's possible so any debugging pointers are welcome. Also, I don't see this problem locally (although, I have to admit that I haven't checked all of the possible runtime combinations).

Here's what I managed to extract so far:

==================================== ERRORS ====================================
_________________ ERROR at setup of test_ssl_adapters[builtin] _________________
[gw1] linux -- Python 3.10.2 /home/runner/work/cheroot/cheroot/.tox/python/bin/python

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f06884add80>
when = 'setup'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.
    
        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

cls        = <class '_pytest.runner.CallInfo'>
duration   = 0.0007801090000043587
excinfo    = <ExceptionInfo AssertionError('previous item was not torn down properly') tblen=6>
func       = <function call_runtest_hook.<locals>.<lambda> at 0x7f06884add80>
precise_start = 297.867987566
precise_stop = 297.868767675
reraise    = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
result     = None
start      = 1644094783.546817
stop       = 1644094783.5475986
when       = 'setup'

.tox/python/lib/python3.10/site-packages/_pytest/runner.py:340: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/python/lib/python3.10/site-packages/_pytest/runner.py:261: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
        ihook      = <_HookCaller 'pytest_runtest_setup'>
        item       = <Function test_ssl_adapters[builtin]>
        kwds       = {}
.tox/python/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
        argname    = 'item'
        args       = ()
        firstresult = False
        kwargs     = {'item': <Function test_ssl_adapters[builtin]>}
        self       = <_HookCaller 'pytest_runtest_setup'>
.tox/python/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
        firstresult = False
        hook_name  = 'pytest_runtest_setup'
        kwargs     = {'item': <Function test_ssl_adapters[builtin]>}
        methods    = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/home/runner/work/cheroot/cheroot/.tox/python/lib/p...pper name='/dev/null' mode='r' encoding='UTF-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>>, ...]
        self       = <_pytest.config.PytestPluginManager object at 0x7f068b7c6b30>
.tox/python/lib/python3.10/site-packages/_pytest/runner.py:156: in pytest_runtest_setup
    item.session._setupstate.setup(item)
        item       = <Function test_ssl_adapters[builtin]>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.runner.SetupState object at 0x7f068a335930>
item = <Function test_ssl_adapters[builtin]>

    def setup(self, item: Item) -> None:
        """Setup objects along the collector chain to the item."""
        needed_collectors = item.listchain()
    
        # If a collector fails its setup, fail its entire subtree of items.
        # The setup is not retried for each item - the same exception is used.
        for col, (finalizers, exc) in self.stack.items():
>           assert col in needed_collectors, "previous item was not torn down properly"
E           AssertionError: previous item was not torn down properly

col        = <Module test_server.py>
exc        = None
finalizers = [<bound method Node.teardown of <Module test_server.py>>]
item       = <Function test_ssl_adapters[builtin]>
needed_collectors = [<Session cheroot exitstatus=<ExitCode.OK: 0> testsfailed=0 testscollected=157>, <Package test>, <Module test_ssl.py>, <Function test_ssl_adapters[builtin]>]
self       = <_pytest.runner.SetupState object at 0x7f068a335930>

.tox/python/lib/python3.10/site-packages/_pytest/runner.py:489: AssertionError

This is what tox has installed in the env that crashes:

python installed: attrs==21.4.0,certifi==2021.10.8,cffi==1.15.0,chardet==4.0.0,charset-normalizer==2.0.11,cheroot @ file:///home/runner/work/cheroot/cheroot/.tox/.tmp/package/2/cheroot-8.6.1.dev216%2Bgfd10ba99-py2.py3-none-any.whl,colorama==0.4.4,coverage==6.2,cryptography==36.0.1,docopt==0.6.2,execnet==1.9.0,idna==3.3,iniconfig==1.1.1,jaraco.context==4.1.1,jaraco.functools==3.5.0,jaraco.text==3.7.0,more-itertools==8.12.0,packaging==21.3,pluggy==1.0.0,portend==3.1.0,py==1.11.0,pycparser==2.21,pyOpenSSL==22.0.0,pyparsing==3.0.7,pypytools==0.6.2,pytest==7.0.0,pytest-cov==2.12.0,pytest-forked==1.4.0,pytest-mock==3.7.0,pytest-rerunfailures==10.2,pytest-sugar==0.9.4,pytest-watch==4.2.0,pytest-xdist==2.5.0,pytz==2021.3,requests==2.27.1,requests-toolbelt==0.9.1,requests-unixsocket==0.3.0,six==1.16.0,tempora==5.0.1,termcolor==1.1.0,tomli==2.0.0,trustme==0.9.0,urllib3==1.26.8,watchdog==2.1.6

plugins: xdist-2.5.0, forked-1.4.0, mock-3.7.0, sugar-0.9.4, rerunfailures-10.2, cov-2.12.0

I've seen this log on GHA under Ubuntu 20.04 and 18.04. Can't say about other envs because they've gotten canceled (will need to check separately).

I don't have a stable repro, unfortunately, but here's CI logs: https://github.com/cherrypy/cheroot/runs/5079935409.

[bluetech]

I know pytest-rerunfailures has some interactions with SetupState that might be causing such errors. I've meant to look into it but haven't got to it. If it's indeed enabled, can you try disabling it and seeing if the problem still reproduces?

[webknjaz]

It is enabled, yes. I think there's a test or two that are decorated. I will try tomorrow, it's past midnight here :)

[The-Compiler]

FWIW no issues so far with my project (qutebrowser) which uses pytest-rerunfailures.

[cnsnyder]

I've seen the same as $subject recently. So far I've narrowed it down to something to do with pytest-forked. At least in my case, when one or more tests are run forked the next test to run fails with $subject. It is also worth mentioning that this strange error only happens for me when the tests are run in a container.

Happy to provide more details if anyone wants! I'm curious what would happen if you temporarily did not use forked.

[bluetech]

It seems all affected use pytest-forked. So it's probably that, not rerunfailures.

• cherrypy/cheroot - uses pytest.mark.forked
• @cnsnyder (do you use @pytest.mark.forked in particular?)
• learning-at-home/hivemind - uses @pytest.mark.forked

[arr-ee]

just a heads up: I did some digging and isolated the issue, alongside with finding a very low-effort workaround: https://github.com/pytest-dev/pytest-forked/issues/67#issuecomment-1964718720

If anyone has any ideas on how to sync SetupState changes from the subprocess without moving setup/teardown into main process (which sounds like a can of worms tbh), please comment!

[RonnyPfannschmidt]

One of the key reasons why forked is unmaintained is that setup state is a can of worms for it ever since scopes,but also then some

[arr-ee]

Wouldn't changing the stage we customise from entirety of pytest_runtest_process to pytest_runtest_call allow us to sidestep caring about setup state management at all? Setup/teardown that is reliant on being ran in a subprocess will break, but this can be worked around by, well, not doing that/moving setup/teardown inside test code (unfortunate but oh well).

[RonnyPfannschmidt]

Possibly, I'm not invested in trying that

Personally I hope to eventually just use xdist primitives to implement sane/safe process isolation of tests

I'm not going to work out how to fix forked

[luoyq-Rockey]

hi，Is there any solution to this problem? I also encountered the same error when using pytest7.1.2，Please give me some advice, thank you very much

[devashish2203]

We use pytest-parallel and are seeing multiple random failures due to the same issue. Is there any recommendation or workaround to this issue?

[webknjaz]

We use pytest-parallel and are seeing multiple random failures due to the same issue. Is there any recommendation or workaround to this issue?

No recommendation. The fix is probably the same — move all the pytest run stages into forks or something. But I see that this project is also archived and will never get any updates unless someone forks it.

[RonnyPfannschmidt]

pytest-parallel heavily hacks around in pytest internals - anything that breaks with it should be brought up to it