robotframework
Data driven tests (Template) and SKIP
In the documentation, it is written that :
Templated tests are also special so that all the rounds are executed even if one or more of them fails. It is possible to use this kind of continue on failure mode with normal tests too, but with the templated tests the mode is on automatically.
In particular : all the rounds are executed even if one or more of them fails
However, if one is skipped, none the following rounds are not executed.
I don't know if it was intended or not but it caused some problems on my tests.
Example:
*** Test Cases ***
Eat yellow fruits
[Documentation] Eat only yellow fruits, Fails if item given is a fruit which is not yellow and skip if item is not a fruit
[Template] Check fruit and eat it
red_apple # FAIL
banana # PASS
plane # SKIP => not a fruit
lemon # should be PASS but is not executed
The documentation is from the time when the SKIP status didn't exist. I believe we didn't really think about this when implementing SKIP, but I have a vague feeling this has been discussed at least once afterwards. I kind of agree it would make sense to run all iterations in this case, but then again explicitly skipping an iteration can be considered to mean "skip remaining iterations".
If we agree to run all iterations also after SKIP, we need to decide what status the test should have if some iterations FAIL. Nowadays an iteration ending with SKIP after an iteration with FAIL gives the test the SKIP status. I believe FAIL after SKIP should behave the same way, but we could also consider changing the overall logic so that any failed iteration gives the test the FAIL status. That would match how any failed test gives a suite the FAIL status regardless of the status of the other tests. Then again, SKIP in test teardown overrides possible earlier FAIL.
I think it would be good to hear opinions from others regarding to these semantics. I'll start a thread about this on the #robotframework channel on our Slack.
If we'd change semantics so that tests get the FAIL status if any iteration fails, regardless are some iterations skipped or not, we should also decided how to handle a case where some iterations pass and some are skipped. I think logically then the test status should be PASS. The test would then get the SKIP status only if all iterations are skipped. This is the same logic we use when deciding the suite status.