swiftlang
Add failure summary section to console output for easier debugging
When running a test suite with multiple failures, it can be difficult to quickly locate and review all the failures, especially when:
- The test suite is large with many tests
- Output is verbose with passing tests interspersed with failures
- Failures are scattered throughout the output
- Multiple expectations fail within the same test Currently, users need to scroll through potentially hundreds or thousands of lines of output to find each failure individually. This makes debugging and triaging test failures more time-consuming than necessary.
Example current output:
✓ Test "successful test 1" passed (0.001s)
✓ Test "successful test 2" passed (0.001s)
✗ Test "failing test" recorded an issue at MyTests.swift:42:5
Expectation failed: (actual → 5) == (expected → 3)
✓ Test "successful test 3" passed (0.001s)
... (hundreds more tests)
✗ Test "another failure" recorded an issue at OtherTests.swift:89:7
Expectation failed: (value → "wrong") == ("correct")
... (more tests)
Proposed solution: Add a failure summary section at the end of the test run that aggregates all failures in one place for quick review. This summary would appear after the standard test results but before the final statistics.
Example proposed output:
✓ Test "successful test 1" passed (0.001s)
✓ Test "successful test 2" passed (0.001s)
✗ Test "failing test" recorded an issue at MyTests.swift:42:5
Expectation failed: (actual → 5) == (expected → 3)
✓ Test "successful test 3" passed (0.001s)
... (hundreds more tests)
FAILED TEST SUMMARY (2 failures)
[1/2] MyModule / MyTests / failing test
✗ Expectation failed: (actual → 5) == (expected → 3)
at MyTests.swift:42:5
[2/2] OtherModule / OtherTests / another failure
✗ Expectation failed: (value → "wrong") == ("correct")
at OtherTests.swift:89:7
Test run completed in 10.5s (pass: xxx, fail: x, skip: x)
Related work: This builds on the experimental AdvancedConsoleOutputRecorder work but integrates the failure summary feature directly into the existing ConsoleOutputRecorder to provide immediate value without waiting for the full hierarchical display feature.
This feature has been discussed with @stmontgomery who suggested bringing the failure summary to the existing console recorder rather than waiting for the experimental advanced recorder to be fully ready. The goal is to provide immediate value to users struggling to find failures in large test suites.
Is it a feature request if you're the one working on it? 🤔
Yes, but Stuart specifically asked me to create the issue to collect feedback from the community before I start implementing. Stuart wants this feature implemented in the current/existing ConsoleOutputRecorder, not the experimental AdvancedConsoleOutputRecorder. His main goal is to quickly bring this feature to users rather than waiting for the full advanced output to be ready. He emphasized that the design should look natural and consistent with what we already have - the current reporter is very simple, just a stream of text lines with no empty lines or breaks.
I'll reference some code from AdvancedConsoleOutputRecorder for the failure tracking logic, but the UI/output needs to be much simpler and match the existing recorder's style and we'd love to hear more about the output idea.