flank icon indicating copy to clipboard operation
flank copied to clipboard

Published 20 hours ago • verified publisher icon Flank

Test case based retries

Open bootstraponline opened this issue 5 years ago • 6 comments

Add a retry mode to flank that retries only failed flaky tests. FTL retry will retry entire shards, including successful tests.

This feature is not currently on the FTL roadmap.

bootstraponline avatar May 12 '20 00:05 bootstraponline

Would love to have this feature!

narenmanoharan avatar May 26 '20 18:05 narenmanoharan

Oh it would be really very useful!

kbazleva avatar Jun 08 '20 14:06 kbazleva

hi guys, do you have plans to implement this feature? It would be nice to re-run only failed tests instead of entire shard.

mariuszmachniakzendesk avatar Aug 25 '20 13:08 mariuszmachniakzendesk

we do plan to support this feature! there are other tasks (bug fixes, stability) that have taken priority over feature work.

bootstraponline avatar Aug 25 '20 14:08 bootstraponline

One thing that would be worth to keep in mind, since FTL does not support such logic it would require from flank creating new matrix (matrices) for failed tests. It's not something extremely difficult but there is another overhead. FTL will set up new device(s) and that takes some time. I think it's not a blocker but let's don't forget about it.

pawelpasterz avatar Sep 21 '20 10:09 pawelpasterz

please also make sure that the total test count tally is not impacted by the retry. If we have 100 tests that execute, and 2 are flaky and get re-ran twice, we should still report 100 tests. I saw this issue in FTL where they actually incremented the total test count, which threw off our jenkins test analyzer.

Joseph-Labonte avatar Oct 02 '20 19:10 Joseph-Labonte