Pytest does not show inner exceptions in PEP-654 `ExceptionGroup`s

[Zac-HD]

Sometimes, you might have code which needs to raise multiple exceptions - for example if you retried an operation, had an error in async or cleanup code, or found multiple bugs with Hypothesis. This is pretty awkward though, and in current Pythons usually means losing some information, or at least making it less accessible.

PEP 654 adds "Exception Groups" as a standard "container" exception, to enable standardised handling of... groups of exceptions. (and except*, which Pytest can ignore)

I therefore propose that we should think about how to display a BaseExceptionGroup, with the potential tree of messages and tracebacks. This isn't urgent, since they'll be released in Python 3.11, but ideally we could support a backport sooner to enable downstream support in Hypothesis, pytest-trio, etc.

[bluetech]

For internal usage, see #8217.

[agronholm]

Just mentioning here too that there is a backport available now.

[Zac-HD]

We are now very close to the release of 3.11.0b1, and there's a backport in increasingly widespread use. I've therefore upgraded this to a bug, because inner exceptions are not displayed at all by default 😬

from exceptiongroup import ExceptionGroup

def f(): raise ValueError("From f()")
def g(): raise RuntimeError("From g()")

def main():
    excs = []
    for callback in [f, g]:
        try:
            callback()
        except Exception as err:
            excs.append(err)
    if excs:
        raise ExceptionGroup("Oops", excs)

def test():
    main()

The native tracebacks work well:

# pytest --tb=native
_________________________________________ test __________________________________________
  + Exception Group Traceback (most recent call last):
  |   File "test.py", line 24, in test
  |     main()
  |   File "test.py", line 20, in main
  |     raise ExceptionGroup("Oops", excs)
  | exceptiongroup.ExceptionGroup: Oops (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "test.py", line 16, in main
    |     call()
    |   File "test.py", line 5, in f
    |     raise ValueError("From f()")
    | ValueError: From f()
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "test.py", line 16, in main
    |     call()
    |   File "test.py", line 9, in g
    |     raise RuntimeError("From g()")
    | RuntimeError: From g()
    +------------------------------------

...but Pytest's default traceback display drops everything under the top-level ExceptionGroup, from tracebacks to error messages and types.

# pytest --tb=auto
_____________________________________ test ______________________________________

    def test():
>       main()

test.py:24:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def main():
        excs = []
        for call in [f, g]:
            try:
                call()
            except Exception as err:
                excs.append(err)
        if excs:
>           raise ExceptionGroup("Oops", excs)
E           exceptiongroup.ExceptionGroup: Oops (2 sub-exceptions)

test.py:20: ExceptionGroup

[Zac-HD]

@pytest-dev/core - I've bounced off this several times now, and could do with suggestions (or better yet a volunteer) to ensure that ExceptionGroup contents are in fact displayed when tests fail 🙏:pray:

I also personally prefer --tb=native and so don't have much of a sense of how we should display ExceptionGroups, though I'd be satisfied with the native display to start with.

[nicoddemus]

Should we split this issue in two, 1) supporting exception groups as if native (already merged), and 2) complete support, meaning showing exception groups in all tb-modes?

I ask because this is scheduled for 7.2, and I believe we are in a satisfactory place for 7.2 with 1) in place.

[asottile]

yeah let's not block 7.2, we haven't released in quite a while and I think this is going to be pretty involved to land

[Zac-HD]

Agreed, and I'd prefer to close this and open a new issue for 'more stylish display of ExceptionGroups' - we do at least display inner errors at all now!

[nicoddemus]

Thanks folks.

I just removed this issue from the 7.2 milestone then.

We can open a new issue, or edit this one to clarify that pytest should honor the traceback mode (instead of falling back to native as it does now). Up to you @Zac-HD!

[Zac-HD]

Closing this one; I'll leave opening the new one to someone with an opinion on what we should do next 😅