subtests.test cannot be used inside a generator

[pmeier]

Imagine I have similar test case setup for multiple tests and I want to use subtests:

import pytest


@pytest.fixture
def create_test_cases(subtests):
    def fn(n):
        for i in range(n):
            with subtests.test(msg="custom message", i=i):
                yield i

    return fn


def test_foo(create_test_cases):
    for i in create_test_cases(5):
        assert i % 2 == 0


def test_bar(create_test_cases):
    for i in create_test_cases(5):
        assert i % 3 == 0

This gives the following output

main.py ,F,F                                                       [100%]

================================ FAILURES =================================
________________________________ test_foo _________________________________
Traceback (most recent call last):
  File "/home/user/main.py", line 24, in test_foo
    assert i % 2 == 0
AssertionError: assert (1 % 2) == 0
________________________________ test_bar _________________________________
Traceback (most recent call last):
  File "/home/user/main.py", line 29, in test_bar
    assert i % 3 == 0
AssertionError: assert (1 % 3) == 0
========================= short test summary info =========================
FAILED main.py::test_foo - assert (1 % 2) == 0
FAILED main.py::test_bar - assert (1 % 3) == 0
============================ 2 failed in 0.03s ============================

As you can see, although the subtests.test context manager is in place, the execution stops after the first failure. Since it is a FAILED instead of a SUBFAILED, one also misses out on the extra information the sub failure would print.

Digging into the code, the problem is

https://github.com/pytest-dev/pytest-subtests/blob/90df760933897a3089b68fe19bed0185019ee11a/pytest_subtests.py#L168-L171

not handling the GeneratorExit.

[nicoddemus]

Hi @pmeier,

Thanks for the detailed report, appreciate it!

If you have the time, please consider opening a pull request. 👍

[pmeier]

I'll send a PR if I figure out how this can be solved. My current understanding is that first the GeneratorExit is raised and only afterwards the actual error. Thus, I think we need more than one context manager. Whether this has to be in user code or can live in subtests.test has yet to be determined.

FWIW, unittest.TestCase.subTest also cannot handle this:

import unittest


class TestFoo(unittest.TestCase):
    def gen(self, n):
        for i in range(n):
            with self.subTest(msg=str(i)):
                yield i

    def test_bar(self):
        for i in self.gen(5):
            assert i % 2 == 0

$ python -m unittest main.py
Exception ignored in: <generator object TestFoo.gen at 0x7f48747bf150>
RuntimeError: generator ignored GeneratorExit
F
======================================================================
ERROR: test_foo (main.TestFoo) [1]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/main.py", line 33, in gen
    yield i
GeneratorExit

======================================================================
FAIL: test_foo (main.TestFoo)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/main.py", line 37, in test_foo
    assert i % 2 == 0
AssertionError

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1, errors=1)

[pmeier]

If I'm reading this right

Raises a GeneratorExit at the point where the generator function was paused. [...] If the generator yields a value, a RuntimeError is raised.

one cannot yield values after the first failure. This of course would completely defeat the purpose of a subtest.

[pmeier]

My best guess is that we are hitting a language limitation here:

import contextlib


def gen(n):
    for i in range(n):
        print(f"Before yield {i}")
        with contextlib.suppress(GeneratorExit):
            yield i
        print(f"After yield {i}")


for i in gen(3):
    print(f"Processing {i}")
    break

Before yield 0
Processing 0
After yield 0
Before yield 1
Exception ignored in: <generator object gen at 0x7ffb35f19550>
RuntimeError: generator ignored GeneratorExit

So, after the first GeneratorExit was raised and even if we catch it, we can no longer yield anything new. As soon as you try, you'll get a RuntimeError.

[RonnyPfannschmidt]

its certainly a language "limitation", as its a intentionally NOT supported pattern

and structurally its absolutely valid, we should go as far as letting subtest special case generator exit and taising a ProgrammingError, as structurally, the generator is intentionally disconnected from exceptions in the outer loop body, outer exception -> generator close

@nicoddemus i propose ensuring subtests transfer generator-exits as is + triggering a warning for the subtest/test

[pmeier]

i propose ensuring subtests transfer generator-exits as is + triggering a warning for the subtest/test

I can send a PR for that. What is the preferred way to warn users from inside a pytest.plugin? warnings.wan("foo", UserWarning)?