python-pytest-harvest
python-pytest-harvest copied to clipboard
smarie
EOFError during pytest session finish with pytest_xdist plugin
cmd
pytest tests/test_model_follow_character_description.py -n 4
Env
- OS: Linux
- Python Version: 3.12.3
- pytest Version: 8.1.1
- pytest Plugins:
- anyio-4.3.0
- asyncio-0.23.6
- xdist-3.6.1
- repeat-0.9.3
- deepeval-0.21.36
- harvest-1.10.5
Error Traceback
============================================= test session starts ==============================================
platform linux -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0
rootdir: /mnt/raid/fran/kai
configfile: pytest.ini
plugins: anyio-4.3.0, asyncio-0.23.6, xdist-3.6.1, repeat-0.9.3, deepeval-0.21.36, harvest-1.10.5
asyncio: mode=Mode.STRICT
initialized: 4/4 workers/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/deepeval/__init__.py:42: UserWarning: You are using deepeval version 0.21.36, however version 0.21.42 is available. You should consider upgrading via the "pip install --upgrade deepeval" command.
warnings.warn(
/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/deepeval/__init__.py:42: UserWarning: You are using deepeval version 0.21.36, however version 0.21.42 is available. You should consider upgrading via the "pip install --upgrade deepeval" command.
warnings.warn(
/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/deepeval/__init__.py:42: UserWarning: You are using deepeval version 0.21.36, however version 0.21.42 is available. You should consider upgrading via the "pip install --upgrade deepeval" command.
warnings.warn(
/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/deepeval/__init__.py:42: UserWarning: You are using deepeval version 0.21.36, however version 0.21.42 is available. You should consider upgrading via the "pip install --upgrade deepeval" command.
warnings.warn(
4 workers [4 items]
.... [100%]Running teardown with pytest sessionfinish...
Traceback (most recent call last):
File "/mnt/raid/fran/miniconda3/bin/pytest", line 8, in <module>
sys.exit(console_main())
^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/_pytest/config/__init__.py", line 197, in console_main
code = main()
^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/_pytest/config/__init__.py", line 174, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
raise exception.with_traceback(exception.__traceback__)
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/_pytest/main.py", line 332, in pytest_cmdline_main
return wrap_session(config, _main)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/_pytest/main.py", line 320, in wrap_session
config.hook.pytest_sessionfinish(
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_callers.py", line 182, in _multicall
return outcome.get_result()
^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
raise exc.with_traceback(exc.__traceback__)
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
teardown.throw(outcome._exception)
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/_pytest/logging.py", line 871, in pytest_sessionfinish
return (yield)
^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
teardown.throw(outcome._exception)
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/_pytest/terminal.py", line 866, in pytest_sessionfinish
result = yield
^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
teardown.throw(outcome._exception)
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/_pytest/warnings.py", line 140, in pytest_sessionfinish
return (yield)
^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/kai/tests/conftest.py", line 82, in pytest_sessionfinish
session_results_df = get_session_results_df(session)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pytest_harvest/plugin.py", line 246, in get_session_results_df
possibly_restore_xdist_workers_structs(session_or_request)
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pytest_harvest/plugin.py", line 476, in possibly_restore_xdist_workers_structs
workers_saved_material = session.config.hook.pytest_harvest_xdist_load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/raid/fran/miniconda3/lib/python3.12/site-packages/pluggy/_manager.py", line
I found out that cloudpickle can solve this issue. I recommend the library will switch to it :) cc @smarie
# conftest.py
import cloudpickle # type: ignore
from typing import OrderedDict, Dict, Any
from logging import warning
from pathlib import Path
from shutil import rmtree
# ## Xdist harvesting using cloudpickle instead of pytest-harvest's default pickling
# ## This is useful when the harvested objects are not serializable by pytest-harvest's default pickling.
RESULTS_PATH = Path('./.xdist_harvested/')
def pytest_harvest_xdist_init() -> bool:
if RESULTS_PATH.exists():
rmtree(RESULTS_PATH)
RESULTS_PATH.mkdir(exist_ok=False)
return True
def pytest_harvest_xdist_worker_dump(worker_id: str, session_items: Any, fixture_store: OrderedDict[Any, Any]) -> bool:
with open(RESULTS_PATH / f'{worker_id}.pkl', 'wb') as f:
try:
cloudpickle.dump((session_items, fixture_store), f)
except Exception as e:
warning(f"Error while pickling worker {worker_id}'s harvested results: [{e.__class__}] {e}")
return True
def pytest_harvest_xdist_load() -> Dict[str, Any]:
workers_saved_material = dict()
for pkl_file in RESULTS_PATH.glob('*.pkl'):
wid = pkl_file.stem
with pkl_file.open('rb') as f:
workers_saved_material[wid] = cloudpickle.load(f)
return workers_saved_material
def pytest_harvest_xdist_cleanup() -> bool:
rmtree(RESULTS_PATH)
return True
Hi both, thanks a lot @junuMoon for finding this issue and @AlmogBaku for finding a solution !
I cannot reproduce the issue on my side, so @junuMoon can you please
- run your code again and confirm that the issue still exists with the latest version of pytest-harvest and pytest-xdist ?
- and if it still exists, try to checkout and install branch #75 and confirm that it solves the issue for you ?
Thanks a lot !
@junuMoon or @AlmogBaku any update on this ?
- can you reproduce the initial issue still ?
- if so, does PR #75 fix it for you ?