Ignored steps in slim are not counted; Junit reports ignored test pages as passed

[tcnh]

Observed behaviour When running a test where some steps are not executed (i.e. when a StopTestException is thrown), Slim reports these steps as ignored, resulting in 'Test not run' in a span with class 'ignore'. The ignore count, however is always 0.

When running such tests through Junit, these tests are always reported as passed. Even if nothing was executed, this happens if a SuiteSetUp throws a stoptestexception (0 asserts, 0 exceptions, but ignored steps).

Expected behaviour If steps are ignored, I'd expect the ignored count to be updated. If the Junit runner encounters a test with no exceptions, assertions o failures, but skipped steps, I's expect the test to be reported as failed (ideally: ignored, but this currently seems hard/impossible to implement in a sustainable matter because we cannot determine that the test is ignored before starting it. That means testStarted has been fired and most listeners will expect a testFinished, that in the current situation marks the test as passed if no failure was fired.

Proposed solution

• Count ignores
• If there are no asserts, no exceptions, but there are ignored steps, report the test as failed with a message describing that everything in the test was ignored. This behaviour is similar to the assertion that is added in FitNesseRunner.executeTests that a run has at least 1 assert/exception.

See: https://github.com/unclebob/fitnesse/pull/1270

[fhoeben]

Sometimes test pages are created that have no assertions intentionally (e.g. they only use a script table to show some information about the system under tests). By changing the behaviour as you suggest suites containing such a test would be counted as failures, while actually nothing failed.

Would it not be better to count the page (e.g. suite setup throwing the exception) as a failure/error and using that to fail the suite, instead of marking other pages as failures? I believe that would also make troubleshooting easier, more clear, as the page that encountered the error is marked as failed and the others as ignored.

[tcnh]

Actually, that's why I added the count for ignored steps. A show would not count as ignored, so these would still be reported as passed.

[fhoeben]

And a check with an empty last column effectively turning the row into a show (I believe those are also rendered as 'ignored')?

Just to make sure I understand the current behaviour: is the SuiteSetUp throwing the exception reported as error (or failed). That seems the most important to me and that should make the whole suite/run be reported as failed.

[tcnh]

You're right. A check with an empty expectation counts as an ignore.. :(