Rerun flaky tests after the others

[hugovk]

If a flaky test fails, it looks like it's rerun immediately.

For flaky integration tests which run against a third party, the external service may only be temporarily down. Running the same test immediately won't help much.

But deferring it, and sticking it at the end of the queue so it's run after all the remaining tests, might give the external service enough time to sort itself out and let the flaky test pass.

I'm seeing this now (using pytest). A flaky test passes in one Travis job, but fails in the next.

Is there an option for deferring flaky tests until after the rest of the test suite?

If not, would it be possible to add one?

Thank you!

[kmike]

+1 to this proposal.

[Jeff-Meadows]

This seems like a reasonable enhancement. Perhaps a command line flag --flaky-defer-reruns or something like that. I wonder if reruns on success (with the min_passes flag set above 1) should also be deferred?

[hugovk]

@Jeff-Meadows It could be argued reruns on success should also be deferred, but if it passes once now, perhaps it's more likely to pass again right now too -- why wait for the third party to get messed up :) And then if a rerun fails, defer the next rerun to give it time to straighten things out.

I'm not using rerun on success though, so don't have strong feelings on this.

Thanks!

[kmike]

I agree with @hugovk; I'm also not using reruns on success, and not going to use them.

[dmakhno]

Cool, can you please suggest where to dig? I'm going to use similar to support setUpClass and tearDownClass.