Unable to filter tests (--test) when re-running failed tests (--rerunfailed)

[ghost]

Filtering tests when rerunning failed tests does not work.

The attached file contains two tests that will always fail.

$ pybot --version
Robot Framework 3.0.2 (Python 2.7.14 on linux2)

$ pybot --output output_full.xml --log NONE --report NONE one.robot.txt
==============================================================================
One.Robot
==============================================================================
HDFS Test                                                             | PASS |
------------------------------------------------------------------------------
Hive Test                                                             | PASS |
------------------------------------------------------------------------------
Abracadabra Test                                                      | FAIL |
AssertionError
------------------------------------------------------------------------------
Alakazam Test                                                         | FAIL |
AssertionError
------------------------------------------------------------------------------
One.Robot                                                             | FAIL |
4 critical tests, 2 passed, 2 failed
4 tests total, 2 passed, 2 failed
==============================================================================
Output:  /home/richcocoa/Downloads/output_full.xml

$ pybot --rerunfailed output_full.xml --test '*alakazam*' --output output.xml --log NONE --report NONE one.robot.txt 
==============================================================================
One.Robot                                                                     
==============================================================================
Abracadabra Test                                                      | FAIL |
AssertionError
------------------------------------------------------------------------------
Alakazam Test                                                         | FAIL |
AssertionError
------------------------------------------------------------------------------
One.Robot                                                             | FAIL |
2 critical tests, 0 passed, 2 failed
2 tests total, 0 passed, 2 failed
==============================================================================
Output:  /home/richcocoa/Downloads/output.xml

A regular run while only filtering test cases works fine (without --rerunfailed):

$ pybot  --test '*alakazam*' --output output.xml --log NONE --report NONE one.robot.txt 
==============================================================================
One.Robot                                                                     
==============================================================================
Alakazam Test                                                         | FAIL |
AssertionError
------------------------------------------------------------------------------
One.Robot                                                             | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output:  /home/richcocoa/Downloads/output.xml

one.robot.txt

[pekkaklarck]

--rerunfailed works so that it picks all failed tests into execution. If you use --test with it, those tests are added to the execution as well. In other words, everything seems to work as designed. How did you expect these options to work together?

[ghost]

I expected --test to further filter the failed tests instead of independently adding more tests. That is, in the above example, I expected only Alakazam Test to be executed when using --test "*alakazam*" with --rerunfailed.

Is there another way to achieve what I want short of writing a custom suite visitor? That is, a way to only execute a subset of failed tests.

[pekkaklarck]

Well, although the current functionality is by design, it doesn't mean the design couldn't be changed. Now that I think about this, I don't see much use cases for --test adding more tests for execution when used with --rerunfailed, but further filtering the selection would probably be useful. The change would be backwards incompatible, but could be done in a major release like RF 3.1 if nobody objects.

If we decide to change this, we probably should consider changing the logic when using --test in combination with --suite as well. Currently both of these options add tests to execution, but we could also change this so that only tests matching --test in a suite matching --suite are selected. This is obviously independent decision, but I'd prefer consistency.

Basically making this change requires two things:

• Asking from users is the current functionality with --rerunfailed and --test (and with --suite and --test) important for them or could it be changed. I would say asking on robotframework-users mailing list and on the Slack community would be enough.
• Someone needs to provide a pull request.

Those who'd want to get this functionality now ought to be able to implement a custom --prerunmodifier. I guess normal --rerunfailed could be used and the modifier could just drop tests that are not needed. These modifiers are used after filtering otherwise, so tests not to be re-run are already dropped.

[yahman72]

I can only agree to this one: "I don't see much use cases for --test adding more tests for execution when used with --rerunfailed" --> in the past 5 years or so and I have never had nor have I heard that anybody in my teams had a use case like this (i.e. "add tests to re-execution"). The use case has always been "exclude tests from re-execution".

[pekkaklarck]

The current when using --test and --rerunfailed together is inconsistent with how --include and --rerunfailed work together. When using --test with --rerunfailed, the effect is cumulative and all failed tests are run along with tests matching --test. On the other hand, when using --include with --rerunfailed, only failed tests that match --include are run.

As mentioned above, the main reason for the current behavior is that --rerunfailed is implemented so that it internally uses --test with each failed test. Thus when using --test separately, there just are more tests to be run. This could be fixed by re-implementing --rerunfailed (and --rerunfailedsuites) so that they filter the original suite separately and don't use --test (or --suite) internally. Then the separately used --test could further filter the resulting suite.

[pekkaklarck]

Care must be taken when changing the behavior of these options. In RF 7.0 we tried to make --suite, --test and --include cumulative (#4721) but that failed miserably. We noticed that making --suite cumulative with --test and --include isn't a good idea already during development and that functionality was never released. We did make --test and --include cumulative, but that caused backwards incompatibility problems and the behavior was reverted in RF 7.0.1 (#5023). Majority of those problems were actually related to --rerunfailed that currently uses --test internally. If we'd re-implement it and made it independent from --test, we could even consider making --test and --include cumulative again.