pytest -x doesn't break on first failing subtest

[pganssle]

I have noticed that when running pytest -x (i.e. stop after the first failure) on a test with failing subtests, the entire test will run - the test won't abort after the first failing subtest. For example, with a test like this:

import pytest

def test_with_subtests(subtests):
    for i in range(10):
        with subtests.test(msg="Message", i=i):
            assert i == 0

The short test summary for pytest -x shows 9 failures instead of 1:

$ pytest -x
(tmp) [/tmp/tmp.1B06JKxL9V]$ pytest -x tests.py
================================================== test session starts ===================================================
platform linux -- Python 3.8.1, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /tmp/tmp.1B06JKxL9V
plugins: subtests-0.3.0
collected 1 item                                                                                                         

tests.py .FFFFFFFFF.                                                                                               [100%]

< skipping failures >
================================================ short test summary info =================================================
FAILED tests.py::test_with_subtests - assert 1 == 0
FAILED tests.py::test_with_subtests - assert 2 == 0
FAILED tests.py::test_with_subtests - assert 3 == 0
FAILED tests.py::test_with_subtests - assert 4 == 0
FAILED tests.py::test_with_subtests - assert 5 == 0
FAILED tests.py::test_with_subtests - assert 6 == 0
FAILED tests.py::test_with_subtests - assert 7 == 0
FAILED tests.py::test_with_subtests - assert 8 == 0
FAILED tests.py::test_with_subtests - assert 9 == 0
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 9 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================== 9 failed, 1 passed in 0.08s ===============================================

I've been using pytest-subtests quite extensively with the reference implementation for PEP 615, and some tests have dozens of subtests (e.g. this one, where the subtest is basically used for parametrization), and I'd like to be able to quickly stop the tests and get short feedback if something is broken, but instead I get dozens of failure messages.

I assume there's a related but possibly trickier question of when you set --max-failures to some number other than 1, are you counting subtests or are you counting top-level tests, but I only ever use -x to mean "stop immediately on first failure", and this is defeating my intuitions on that use case.

[Celeborn2BeAlive]

Same issue here, it makes it quite hard to iterate on a single large test having many small subtests (most of my subtests are just a single assert line)

[rhoban13]

I was tinkering with how one might implement this hoping I might contribute, but honestly I'm not sure how best to proceed.

When the subtests.test fails, the value of the session.shouldfail gets set when we call pytest_runtest_logreport here. However since that doesn't raise anything the way it does during the main test loop here, the test body continues and the next subtests.test runs. We could certainly raise that same exception right there from inside the context managed subtests.test, however the resulting stack trace & output really looks gross as it now includes the trace from this cleanup step as well. In general I'm not sure how best to raise anything from after that yield.

If anyone has ideas I'm willing to give them a try.

[rhoban13]

It looks like I can call pytest.exit from inside that handling and it actually looks like the stack and output I'd somewhat hope to see. I've not used pytest.exit before, so not certain if there's some implications I'd need to think through first or if this approach might work.

[nicoddemus]

however the resulting stack trace & output really looks gross as it now includes the trace from this cleanup step as well

Perhaps using __tracebackhide__ can help with that (see here)? Calling pytest.exit does not seem right to me.