swift-testing icon indicating copy to clipboard operation
swift-testing copied to clipboard
verified publisher icon swiftlang

Add failure summary to ConsoleOutputRecorder

Open tienquocbui opened this issue 4 months ago • 1 comments

Overview

This PR implements a failure summary section for ConsoleOutputRecorder to help users quickly locate and review test failures in large test suites.

Closes: #1355

Architectural Changes Implemented

  1. State Management (Critical Fix)
  • [x] Uses the existing single Lock from HumanReadableOutputRecorder
  • [x] Fits naturally with existing Graph traversal logic
  1. Data Structure
  • [x] Added lightweight TestData.IssueInfo struct
  • [x] Each issue now has its own source location (not one per test)
  • [x] Stores: sourceLocation, description, isKnown
  • [x] Issues stored as array in TestData.issues
  • [x] testDisplayName and testCaseArguments stored in outer TestData (not repeated per issue)
  • [x] Kept issueCount dictionary separate for efficiency (parallel tracking)
  1. Interface
  • [x] Added generateFailureSummary() method to HumanReadableOutputRecorder
  • [x] Focuses purely on content generation (newlines handled by caller)
  • [x] Traverses Graph to collect all failed tests
  • [x] ConsoleOutputRecorder calls this at runEnded event and adds spacing
  1. Output Improvements
  • [x] Uses indentation with dashes for clarity (- description)
  • [x] Shows fully qualified suite/test path (addresses user complaints about missing suite info)
  • [x] Each issue listed separately with its own source location
  • [x] Blank lines before and after summary for visual separation
  • [x] Uses expandedDebugDescription() for detailed failure output with full type info and expanded values
  • [x] Shows custom display names in quotes (e.g., "Custom Display Name")
  • [x] Displays parameterized test arguments once per test (e.g., (value → 7)) to identify which specific input failed
  • [x] Test metadata stored once per test, not repeated for each issue

Example Output

Proposed Failure Summary

Regular Test Failure:

✗  TestingTests/FailureSummaryDemoTests/MathTests/"Division fails"
  - (result: Swift.Int → 3) == (4: Swift.Int → 4)
    at TestingTests/FailureSummaryDemoTests.swift:26

Parameterized Test Failure (shows which argument failed):

✗  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
  (value → 7)
  - (value % 2: Swift.Int → 1) == (0: Swift.Int → 0)
    at TestingTests/FailureSummaryDemoTests.swift:72

Test with Custom Display Name:

✗  TestingTests/FailureSummaryDemoTests/ParameterizedTests/"This is a custom display name"
  - (value: Swift.Int → 42) == (100: Swift.Int → 100)
    at TestingTests/FailureSummaryDemoTests.swift:83

Complete Test Output

Click to expand full test run output 
(base) buitienquoc@Kelvins-MacBook-Pro swift-testing % swift test --filter "FailureSummaryDemoTests" --disable-xctest
[1/1] Planning build
Building for debugging...
[257/257] Linking swift-testingPackageTests
Build complete! (39.83s)
􀟈  Test run started.
􀄵  Testing Library Version: 6.3-dev (d8b140d780dc2da - modified)
􀟈  Suite "Failure Summary Demo" started.
􀟈  Suite "Math Operations" started.
􀟈  Suite "String Operations" started.
􀟈  Test "Addition works correctly" started.
􀟈  Test "String comparison passes" started.
􀟈  Test "Multiple string failures" started.
􀟈  Test "String concatenation fails" started.
􀟈  Suite "Array Operations" started.
􀟈  Test "Division fails" started.
􀟈  Test "Array equality" started.
􀟈  Suite "Parameterized Tests" started.
􁁛  Test "Addition works correctly" passed after 0.001 seconds.
􁁛  Test "String comparison passes" passed after 0.001 seconds.
􁁛  Test "Array equality" passed after 0.001 seconds.
􀟈  Test "Array contains element" started.
􀟈  Test "This is a custom display name" started.
􀟈  Test "Check even numbers" started.
􀟈  Test "String length validation" started.
􀟈  Test case passing 1 argument value → 4 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 6 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 7 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 10 to "Check even numbers" started.
􀟈  Test case passing 1 argument text → "hello" to "String length validation" started.
􀟈  Test case passing 1 argument text → "world" to "String length validation" started.
􀟈  Test case passing 1 argument text → "a" to "String length validation" started.
􀟈  Test case passing 1 argument value → 2 to "Check even numbers" started.
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:46:7: Expectation failed: "a" == "b"
􀢄  Test "String concatenation fails" recorded an issue at FailureSummaryDemoTests.swift:36:7: Expectation failed: (greeting + name → "HelloWorld") == "Hello World"
􀢄  Test "This is a custom display name" recorded an issue at FailureSummaryDemoTests.swift:83:7: Expectation failed: (value → 42) == 100
􀢄  Test "String length validation" recorded an issue with 1 argument text → "a" at FailureSummaryDemoTests.swift:77:7: Expectation failed: (text.count → 1) >= 5
􀢄  Test "Array contains element" recorded an issue at FailureSummaryDemoTests.swift:57:7: Expectation failed: (numbers → [1, 2, 3, 4, 5]).contains(10)
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:47:7: Expectation failed: ("hello".count → 5) == 10
􀢄  Test "Division fails" recorded an issue at FailureSummaryDemoTests.swift:26:7: Expectation failed: (result → 3) == 4
􀢄  Test "Check even numbers" recorded an issue with 1 argument value → 7 at FailureSummaryDemoTests.swift:72:7: Expectation failed: (value % 2 → 1) == 0
􀢄  Test "This is a custom display name" failed after 0.003 seconds with 1 issue.
􀢄  Test "Array contains element" failed after 0.004 seconds with 1 issue.
􀟈  Test case passing 1 argument text → "swift" to "String length validation" started.
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:48:7: Expectation failed: ("swift".uppercased() → "SWIFT") == "swift"
􀢄  Test "String concatenation fails" failed after 0.005 seconds with 1 issue.
􀢄  Suite "Array Operations" failed after 0.005 seconds with 1 issue.
􀢄  Test "Division fails" failed after 0.005 seconds with 1 issue.
􀢄  Test "String length validation" with 4 test cases failed after 0.004 seconds with 1 issue.
􀢄  Test "Check even numbers" with 5 test cases failed after 0.004 seconds with 1 issue.
􀢄  Suite "Math Operations" failed after 0.005 seconds with 1 issue.
􀢄  Test "Multiple string failures" failed after 0.005 seconds with 3 issues.
􀢄  Suite "Parameterized Tests" failed after 0.005 seconds with 3 issues.
􀢄  Suite "String Operations" failed after 0.005 seconds with 4 issues.
􀢄  Suite "Failure Summary Demo" failed after 0.007 seconds with 9 issues.
􀄵  /// Demo tests showcasing the refactored failure summary feature.

Test run had 7 failed tests with 9 issues:
􀢄  TestingTests/FailureSummaryDemoTests/MathTests/"Division fails"
  - (result: Swift.Int → 3) == (4: Swift.Int → 4)
    at TestingTests/FailureSummaryDemoTests.swift:26
􀢄  TestingTests/FailureSummaryDemoTests/ArrayTests/"Array contains element"
  - (numbers: Swift.Array<Swift.Int> → [1, 2, 3, 4, 5]).contains(10: Swift.Int → 10)
    at TestingTests/FailureSummaryDemoTests.swift:57
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
  (value → 7)
  - (value % 2: Swift.Int → 1) == (0: Swift.Int → 0)
    at TestingTests/FailureSummaryDemoTests.swift:72
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testStringLength(text:)/"String length validation"
  (text → "a")
  - (text.count: Swift.Int → 1) >= (5: Swift.Int → 5)
    at TestingTests/FailureSummaryDemoTests.swift:77
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/"This is a custom display name"
  - (value: Swift.Int → 42) == (100: Swift.Int → 100)
    at TestingTests/FailureSummaryDemoTests.swift:83
􀢄  TestingTests/FailureSummaryDemoTests/StringTests/"Multiple string failures"
  - ("a": Swift.String → "a") == ("b": Swift.String → "b")
    at TestingTests/FailureSummaryDemoTests.swift:46
  - ("hello".count: Swift.Int → 5) == (10: Swift.Int → 10)
    at TestingTests/FailureSummaryDemoTests.swift:47
  - ("swift".uppercased(): Swift.String → "SWIFT") == ("swift": Swift.String → "swift")
    at TestingTests/FailureSummaryDemoTests.swift:48
􀢄  TestingTests/FailureSummaryDemoTests/StringTests/"String concatenation fails"
  - (greeting + name: Swift.String → "HelloWorld") == ("Hello World": Swift.String → "Hello World")
    at TestingTests/FailureSummaryDemoTests.swift:36

􀢄  Test run with 10 tests in 5 suites failed after 0.007 seconds with 9 issues.
(base) buitienquoc@Kelvins-MacBook-Pro swift-testing %

tienquocbui avatar Nov 13 '25 22:11 tienquocbui

Would it make sense to factor the new code into a dedicated TestRunSummary type? (I'm not saying you must, just asking.)

grynspan avatar Nov 14 '25 04:11 grynspan

@grynspan thanks for the suggestion! I discussed that with Stuart and I've refactored the failure summary logic into a dedicated TestRunSummary type. Would you mind taking another look when you get a chance? Thanks!

tienquocbui avatar Nov 18 '25 14:11 tienquocbui