pytest-xdist
pytest-xdist copied to clipboard
Published
20 hours ago •
pytest-dev
pytest-dev
stepwise doesn't stop the suite on first failure with xdist
Stepwise is supposed to behave like -x but this doesn't work with xdist:
You may think of it as a combination of the -x option, which exits the test run after a failing test, and the --lf option from pytest-cache, which only runs failing tests.
graingert@onomastic:~/projects/foo$ cat tests/test_foo.py
import pytest
def test_1():
assert True
def test_2():
assert True
def test_3():
assert False
def test_4():
assert True
def test_5():
assert True
def test_6():
assert True
Running it the first time, the test suite doesn't stop: (no -x behavior)
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [6]
scheduling tests via LoadScheduling
tests/test_foo.py::test_1
[gw0] [ 16%] PASSED tests/test_foo.py::test_1
tests/test_foo.py::test_2
[gw0] [ 33%] PASSED tests/test_foo.py::test_2
tests/test_foo.py::test_3
[gw0] [ 50%] FAILED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 66%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 83%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________________________ test_3 __________________________________________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/graingert/.virtualenvs/pytest/bin/python3.7
def test_3():
> assert False
E assert False
tests/test_foo.py:10: AssertionError
==================================================================================== 1 failed, 5 passed in 0.23 seconds ====================================================================================
Running it the second time, the test suite starts from the first failing test but doesn't stop: (no -x behavior)
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [4]
scheduling tests via LoadScheduling
tests/test_foo.py::test_3
[gw0] [ 25%] FAILED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 50%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 75%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________________________ test_3 __________________________________________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/graingert/.virtualenvs/pytest/bin/python3.7
def test_3():
> assert False
E assert False
tests/test_foo.py:10: AssertionError
==================================================================================== 1 failed, 3 passed in 0.21 seconds ====================================================================================
Ok lets make the test pass:
graingert@onomastic:~/projects/foo$ cat tests/test_foo.py
import pytest
def test_1():
assert True
def test_2():
assert True
def test_3():
assert True
def test_4():
assert True
def test_5():
assert True
def test_6():
assert True
ok it starts from the failure and continues the suite, this is correct
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [4]
scheduling tests via LoadScheduling
tests/test_foo.py::test_3
[gw0] [ 25%] PASSED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 50%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 75%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
========================================================================================= 4 passed in 0.22 seconds =========================================================================================
and when run again the whole suite passes, this is correct too
graingert@onomastic:~/projects/foo$ pytest -vvv --sw -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [6]
scheduling tests via LoadScheduling
tests/test_foo.py::test_1
[gw0] [ 16%] PASSED tests/test_foo.py::test_1
tests/test_foo.py::test_2
[gw0] [ 33%] PASSED tests/test_foo.py::test_2
tests/test_foo.py::test_3
[gw0] [ 50%] PASSED tests/test_foo.py::test_3
tests/test_foo.py::test_4
[gw0] [ 66%] PASSED tests/test_foo.py::test_4
tests/test_foo.py::test_5
[gw0] [ 83%] PASSED tests/test_foo.py::test_5
tests/test_foo.py::test_6
[gw0] [100%] PASSED tests/test_foo.py::test_6
========================================================================================= 6 passed in 0.21 seconds =========================================================================================
note that -x and --lf work correctly with xdist:
graingert@onomastic:~/projects/foo$ cat tests/test_foo.py
import pytest
def test_1():
assert True
def test_2():
assert True
def test_3():
assert False
def test_4():
assert True
def test_5():
assert True
def test_6():
assert True
graingert@onomastic:~/projects/foo$ pytest -vvv -x -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [6]
scheduling tests via LoadScheduling
tests/test_foo.py::test_1
[gw0] [ 16%] PASSED tests/test_foo.py::test_1
tests/test_foo.py::test_2
[gw0] [ 33%] PASSED tests/test_foo.py::test_2
tests/test_foo.py::test_3
[gw0] [ 50%] FAILED tests/test_foo.py::test_3
================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________________________ test_3 __________________________________________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/graingert/.virtualenvs/pytest/bin/python3.7
def test_3():
> assert False
E assert False
tests/test_foo.py:10: AssertionError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================================== 1 failed, 2 passed in 0.27 seconds ====================================================================================
graingert@onomastic:~/projects/foo$ pytest -vvv -x --lf -n 1 tests/
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/graingert/.virtualenvs/pytest/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/graingert/projects/foo/tests, inifile: pytest.ini
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/graingert/projects/foo
[gw0] Python 3.7.3 (default, Apr 3 2019, 05:39:12) -- [GCC 8.3.0]
gw0 [1]
scheduling tests via LoadScheduling
tests/test_foo.py::test_3
[gw0] [100%] FAILED tests/test_foo.py::test_3
================================================================================================= FAILURES =================================================================================================
__________________________________________________________________________________________________ test_3 __________________________________________________________________________________________________
[gw0] linux -- Python 3.7.3 /home/graingert/.virtualenvs/pytest/bin/python3.7
def test_3():
> assert False
E assert False
tests/test_foo.py:10: AssertionError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================================================= 1 failed in 0.27 seconds =========================================================================================
graingert@onomastic:~/projects/foo$
Just hit that, it might actually be better if xdist plain refused to run under stepwise as not only does it not work, it seems to cause strange issues e.g. resuming only runs a small fraction of the tests, and it's not clear that they even were failing ones?
It's a bit of a shame because stepwise is amazing, but if you need xdist to run your test suite in a reasonable time frame...
