kyua icon indicating copy to clipboard operation
kyua copied to clipboard

Do not count skipped as passed in test cmd

Open ihoro opened this issue 1 year ago • 2 comments

It changes only output of 'kyua test' CLI command. Hence, other outputs
like junit are kept intact for CI and other use cases. It's meant to
improve UX of attended use cases.

The issue is that the following can be tricky to interpret:

  222/222 passed (0 failed)

It can be read as all tests are passed, but it might be a summary line
of all tests skipped due to some requirement is not met.

It's reworked to easily distinguish such cases:

  222/222 passed (0 broken, 0 failed, 0 skipped)
  0/222 passed (0 broken, 0 failed, 222 skipped)

The overall formula is:

  <actually passed>/<total> (<details about not actually passed ones>)

ihoro avatar Sep 25 '24 09:09 ihoro

It mirrors https://reviews.freebsd.org/D46653.

ihoro avatar Sep 25 '24 09:09 ihoro

I prefer the reporting that @kprovost suggested in the FreeBSD review. "7/7 passed (and 3 were skipped)" is more immediately obvious to me than "7/10 passed (the three that did not pass were skipped)." Or don't print as a ratio. "7 passed, 0 failed, 3 skipped, 10 total"

emaste avatar Nov 13 '24 20:11 emaste