allure-framework
Missing tests in report after pytest-timeout crashes the python process
I'm submitting a ...
- [x] bug report
- [ ] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.
What is the current behavior?
When the timeout occurred (from pytest-timeout), the python process is stopping brutally (crashes), the current test and the tests that follow after don't appear in the report at all.
If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem
Running 4 tests with timeout of 8 seconds using pytest-timeout, while the 3rd test takes 10 seconds. The execution stops brutally during the 3rd test and the allure report shows only 2 tests that passed.
This is the code of the tests: import time
def test_this(): try: time.sleep(0.1) except Exception: pass
def test_this3(): try: time.sleep(0.1) except Exception: pass
def test_that(): time.sleep(10) pass
def test_this2(): try: time.sleep(0.1) except Exception: pass
What is the expected behavior?
The 3rd and the 4th tests should be in the report at least with (not tested or skipped status)
What is the motivation / use case for changing the behavior?
To avoid misleading that the execution passed successfully and also to keep the running suite with the same number of tests.
Please tell us about your environment:
- Allure version: 2.29.0
- Test framework: [email protected]
- Allure adaptor: [email protected]
Other information
I think you should populate the report the remaining tests and the one that cause the timeout.
- I saw a PR in allure/android that fixed similar issue.
Stacktrace: +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.3\plugins\python-ce\helpers\pycharm\_jb_pytest_runner.py", line 51, in <module>
sys.exit(pytest.main(args, plugins_to_load + [Plugin]))
File "C:\Python37\lib\site-packages\_pytest\config\__init__.py", line 163, in main
config=config
File "C:\Python37\lib\site-packages\pluggy\hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "C:\Python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "C:\Python37\lib\site-packages\_pytest\main.py", line 316, in pytest_cmdline_main
return wrap_session(config, _main)
File "C:\Python37\lib\site-packages\_pytest\main.py", line 269, in wrap_session
session.exitstatus = doit(config, session) or 0
File "C:\Python37\lib\site-packages\_pytest\main.py", line 323, in _main
config.hook.pytest_runtestloop(session=session)
File "C:\Python37\lib\site-packages\pluggy\hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "C:\Python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "C:\Python37\lib\site-packages\_pytest\main.py", line 348, in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "C:\Python37\lib\site-packages\pluggy\hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "C:\Python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 109, in pytest_runtest_protocol
runtestprotocol(item, nextitem=nextitem)
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 126, in runtestprotocol
reports.append(call_and_report(item, "call", log))
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 215, in call_and_report
call = call_runtest_hook(item, when, **kwds)
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 255, in call_runtest_hook
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 311, in from_call
result: Optional[TResult] = func()
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 255, in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "C:\Python37\lib\site-packages\pluggy\hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "C:\Python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 162, in pytest_runtest_call
item.runtest()
File "C:\Python37\lib\site-packages\_pytest\python.py", line 1641, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "C:\Python37\lib\site-packages\pluggy\hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "C:\Python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "C:\Python37\lib\site-packages\_pytest\python.py", line 183, in pytest_pyfunc_call
result = testfunction(**testargs)
File "C:\GitRep\ng_fw_automation\fw_automation\test_try.py", line 24, in test_that
time.sleep(10)
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
Process finished with exit code 1
Well, signal mode works perfectly, but is available on POSIX systems only (and is used by default AFAIK).
On Windows, yes, pytest-timeout uses thread mode and doesn't have other means but to call os._exit to interrupt a hung test, so no chance to do anything after is fires. So, this is not a bug, more of lack of support of this particular plugin.
We may potentially mark the test as failed and the rest as skipped in a custom wrapper over pytest_timeout.timeout_timer function in pytest_timeout_set_timer hook if thread mode is used. I don't like it, because this couples us with pytest-timeout, but since this is an official plugin by pytest-dev team I think it would be more or less ok.
Can't name any date though, as this requires some sort of refactoring (now test results are closed and saved in pytest hooks which are hard to call directly).
