pytest-xdist
pytest-xdist copied to clipboard
pytest-dev
Select which test should be distributed and which test should be run without concurrence
I would like to use pytest-xdist in mne-lsl: https://github.com/mne-tools/mne-lsl/pull/229
However, in the test suite, some test need to be run while no other test are running because they spawn network resources which might be pick-up by other tests. Example:
def test1(...):
spawn_network_resources()
...
def test2(...):
resources = resolve_all_network_resources()
assert len(resources) == 0
spawn_network_resources()
resources = resolve_all_network_resources()
assert len(resources) == 1
In this example, test2 would be affected by test1 run in a different worker.
To avoid this problem, I would like new markers from pytest-xdist to mark which test should not be distributed and should be run through --dist no, either before or after all the other tests distributed to workers.
e.g.
@pytest.mark.no_xdist
def test2(...):
My current alternative idea is to create this marker and to run 2 separate pytest command.
The difference between this request and my alternative lies that developers won't accidently run the test suit with xdist and without the -m "not no_xdist". The -m flag is not needed anymore as xdist will handle dispatching correctly.
When xdist is active, no tests run in the controling process
There's no plan for a meta scheduler that ensures noncoop tests are executed on a single node in isolation
OK thank you! Could that be a feature request then? Some test need to be run in isolation; and preventing an entire test suite to run on xdist because of a minority is disappointing.
If there is no plan/desire to add this feature, feel free to close this issue!
there is a desire to add it, but lack of engineering time to build it
@mscheltienne for me it works adding sequencial tests in a specific group (by mark) and then execute pytest with --dist loadgroup as described here https://pytest-xdist.readthedocs.io/en/latest/distribution.html#running-tests-across-multiple-cpus
If I read this correctly:
--dist loadgroup: Tests are grouped by the xdist_group mark. Groups are distributed to available workers as whole units. This guarantees that all tests with same xdist_group name run in the same worker.
It just means that the tests in loadgroup will be executed by the same worker. However, another worker might be executing other tests at the same time, and this is what I would want to avoid. I wanted to have a group of tests that should run in xdist, and a group of tests that should run sequentially on a single worker.
@mscheltienne yeah pack all the test that you want to execute sequentially in the same worker using the same group, all the rest will be distributed to all the other workers
Yes, but that's my issue, I while the tests run sequentially are running, the other tests are also running simultaneously. I want to split my workflow in 2 steps, one where the distributed tests run, and then one where the sequential tests run. Anyway, for now I'm not going to use xdist.
This would need a new scheduler, currently I also use distinct calls to pytest to run sequential and parallel tests