Detect Flaky tests in an optimized way

[taman9333]

Right now, I am trying to detect flaky tests, but I think looping over the test_results.json file will be inefficient since I would need to go through all packages. However, I realized that the output of the gotestsum command lists the different packages being tested, along with whether each test succeeded or failed. At the end of the output, there is a === Failed section.

For example, the output in the case of the github-actions format might look like this:

::group::PASS cmd.TestStable (0.00s)
    main_test.go:9: This is a stable test

::endgroup::
::group::FAIL cmd.Test_Flaky2 (0.00s)
    main_test.go:21: Flaky test failed!

::endgroup::
  FAIL Package cmd (297ms)

  PASS mathutils.TestAdd (0.00s)
  PASS mathutils.TestMultiply (0.00s)
  PASS mathutils.Test_AddFlaky (0.00s)
  PASS Package mathutils (450ms)


DONE 5 tests, 1 failure in 1.011s

::group::FAIL cmd.Test_Flaky2 (re-run 1) (0.00s)
    main_test.go:21: Flaky test failed!

::endgroup::
  FAIL Package cmd (184ms)


DONE 2 runs, 6 tests, 2 failures in 1.336s

PASS cmd.Test_Flaky2 (re-run 2) (0.00s)
  PASS Package cmd (162ms)

=== Failed
=== FAIL: cmd Test_Flaky2 (0.00s)
    main_test.go:21: Flaky test failed!

=== FAIL: cmd Test_Flaky2 (re-run 1) (0.00s)
    main_test.go:21: Flaky test failed!

DONE 6 runs, 10 tests, 5 failures in 2.546s

I need a way to extract only the === Failed section, if it exists, so that I can loop over the failed tests after this section. This approach would allow me to avoid iterating over all test cases for every package and would help identify flaky tests more quickly, especially when working on a large project with many test cases.

[dnephin]

Thanks for the issue. I'd suggest using --jsonfile argument to write the json to a file. Then you can use https://pkg.go.dev/github.com/gotestyourself/gotestsum/testjson#ScanTestOutput to read that file and then call Failures() to get the list of failures.