pytest-xdist + coverage + intelligent test runner does not work

[hdadkhah1]

Summary of problem

pytest xdist+cov not working with intelligent test runner, the collectors are failing to stop

Which version of dd-trace-py are you using?

2.6.1

Which version of pip are you using?

pip 23.3.1

Which libraries and their versions are you using?

pytest-cov = "^4.1.0" coverage = "^7.4.1" pytest-xdist= "^3.5.0" pytest = "^7.1.2" python = "^3.10"

How can we reproduce your problem?

pytest-xdist + coverage option + intelligent test runner

e.g. DD_CIVISIBILITY_ITR_ENABLED = True DD_CIVISIBILITY_AGENTLESS_ENABLED = True python -m pytest
--ddtrace
--ddtrace-patch-all
--import-mode=importlib
--reruns=5
--durations=50 --maxfail=$(TESTS_MAXFAIL)
--dist=loadscope -p no:pastebin -p no:nose -p no:doctest
--report-log="${TEST_DIR}"/unit-results.jsonl
--cov-config .coveragerc.unit --cov-fail-under 0.00 --cov=./${SRC_PATH}
--cov-append
--cov-report xml:"${TEST_DIR}"/cov_result_unit.xml --junitxml="${TEST_DIR}"/unit-results.xml

What is the result that you get?

INTERNALERROR> def worker_internal_error(self, node, formatted_error): INTERNALERROR> """ INTERNALERROR> pytest_internalerror() was called on the worker. INTERNALERROR> INTERNALERROR> pytest_internalerror() arguments are an excinfo and an excrepr, which can't INTERNALERROR> be serialized, so we go with a poor man's solution of raising an exception INTERNALERROR> here ourselves using the formatted message. INTERNALERROR> """ INTERNALERROR> self._active_nodes.remove(node) INTERNALERROR> try: INTERNALERROR> > assert False, formatted_error INTERNALERROR> E AssertionError: Traceback (most recent call last): INTERNALERROR> E File "/usr/local/lib/python3.10/site-packages/_pytest/main.py", line 270, in wrap_session INTERNALERROR> E session.exitstatus = doit(config, session) or 0 INTERNALERROR> E File "/usr/local/lib/python3.10/site-packages/_pytest/main.py", line 324, in _main INTERNALERROR> E config.hook.pytest_runtestloop(session=session) INTERNALERROR> E File "/usr/local/lib/python3.10/site-packages/pluggy/_hooks.py", line 433, in call INTERNALERROR> E return self._hookexec(self.name, self._hookimpls, kwargs, firstresult) INTERNALERROR> E File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 112, in _hookexec INTERNALERROR> E return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> E File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 133, in _multicall INTERNALERROR> E teardown[0].send(outcome) INTERNALERROR> E File "/usr/local/lib/python3.10/site-packages/pytest_cov/plugin.py", line 298, in pytest_runtestloop INTERNALERROR> E self.cov_controller.finish() INTERNALERROR> E File "/usr/local/lib/python3.10/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper INTERNALERROR> E return meth(self, *args, **kwargs) INTERNALERROR> E File "/usr/local/lib/python3.10/site-packages/pytest_cov/engine.py", line 388, in finish INTERNALERROR> E self.cov.stop() INTERNALERROR> E File "/usr/local/lib/python3.10/site-packages/coverage/control.py", line 663, in stop INTERNALERROR> E self._collector.stop() INTERNALERROR> E File "/usr/local/lib/python3.10/site-packages/coverage/collector.py", line 395, in stop INTERNALERROR> E assert self._collectors[-1] is self, ( INTERNALERROR> E AssertionError: Expected current collector to be <Collector at 0x7f1b37065c30: CTracer>, but it's <Collector at 0x7f1b370663e0: CTracer> INTERNALERROR> INTERNALERROR> /etc/pytest-xdist/src/xdist/dsession.py:197: AssertionError

What is the result that you expected?

no exception

[romainkomorndatadog]

Hi @hdadkhah1 , thanks for the crash report. We did merge #8328 into this release, but that only addresses the cases where the exception happens in our code, and it looks like pytest-xdist introduces some more race conditions as well.

We don't currently support using the Intelligent Test Runner and coverage at the same time because we use it for our coverage collection. Even without the crashes, you would see some gaps in coverage because, in our current implementation, we effectively steal the global coverage object while tests run (even when we're not skipping tests and collecting coverage on the excluded branches). I'll add a note to our docs about these limitations. We do plan to move away from the coverage package for collection, but I don't have a timetable for that yet.

We also have some known issues with pytest-xdist in terms of the quality of data we report, because we over-report the number of sessions, and potentially, modules and suites as well (depending on how tests end up being distributed). That's another thing we plan to address (although here again, I don't have a timetable).

[github-actions[bot]]

This issue has been automatically closed after a period of inactivity. If it's a feature request, it has been added to the maintainers' internal backlog and will be included in an upcoming round of feature prioritization. Please comment or reopen if you think this issue was closed in error.