kiso-testing icon indicating copy to clipboard operation
kiso-testing copied to clipboard

TestSuiteSetup and TestSuiteTaerdown handled like testcases

Open cody82 opened this issue 3 years ago • 1 comments

When defining a BasicTestSuiteSetup or BasicTestSuiteTeardown for a testsuite, those are handled like testcases. This causes a number of issues:

  • The count of succeeded/failed tests at the end of the test run is wrong: E.g. 1 testcase + BasicTestSuiteSetup + BasicTestSuiteTeardown = 3 testcases, but it should display only 1 testcase.
  • If a BasicTestSuiteSetup fails, the testsuite continues to execute, which probably is not intended.

So at least the BasicTestSuiteSetup and BasicTestSuiteTeardown should not be counted as testcases. I am not sure about the other problem.

cody82 avatar May 20 '22 09:05 cody82

The parameter "fail-fast" now exist since 0.16.0, see https://kiso-testing.readthedocs.io/en/latest/whats_new/version_0_16_0.html#fail-fast which will stop your test suite execution

A deeper analysis is needed to check if we can change the testcases runs

sebastianpfischer avatar Jun 03 '22 08:06 sebastianpfischer