Separate test output for each `testing` context

[Limess]

Proposal

Support separating test output to commands ran for each testing context, rather than the whole testing context.

Context

I originally raised this question in Slack (the Slack conversation will eventually expire)

I’m writing tests which each produce a lot of output and was wrapping related (but not co-dependent) tests in deftest and then naming each with testing but that results in the output for everything in the deftest block being added to the test output for each failure.

The current behaviour makes it a lot more difficult to debug the steps for a failure, particularly in integration tests where there is a lot going on. It's possible to only use deftest to

i.e. I wish to do :

(deftest v1-updated-events
  (testing "Something about events..."
  ...
  )
  (testing "Something else about events..."
  ...
  )
)

and receive only the output for executions in the Something about events... testing context when an assertion in that context fails. Currently the test output for a failure shows all executions in the deftest block, even if the other testing blocks were succesful.

Currently in order to get separate captured output for each context you currently need to use a separate deftest block:

(deftest something-about-events
  ...
)
(deftest something-else-about-events
  ...
)

which removes the nested test cases and string names in the test output.

Downsides

It's possible to use testing at any level, e.g. only using it to wrap a single it statement. This could lead to unclear logs if the actual execution isn't captured in the testing block - only the assertion.

[plexus]

Here's the permalink to the slack conversation

I think making this an opt-in, configurable feature is not a bad idea. This would require changes in the output capturing plugin and the reporter code.

The reporter currently looks for ::capture/output on the testable, so it always finds the same output for all assertions in a test. Instead it should first look for ::capture/output on the clojure.test event map.

Then it's a matter of the capture-output plugin implementing a pre-report hook that adds the output to the event map directly when it's a :kaocha/fail-type event, and that resets the output buffer upon :kaocha/end-group events.

[oxalorg]

This feature appears to be beneficial, but it has been some time since we last examined it.

If anyone encounters this issue and requires this feature, please feel free to reopen this ticket. Better yet, you could assist with a submission of a PR based on plexus's comments above.