microsoft
[Feature] Programmatically mark a test for retry during runtime
Short user story: As a Playwright user testing web applications, I need to mark tests for retry during runtime and regardless of their pre-configured retries, in order to retry only for the specific condition(s) that require it. I do not wish to configure global or per-test retries prior to runtime to meet this need, as that will inevitably result in retries for reasons outside of these specific conditions.
Real world explanation: Our stack uses Caddy which has a issue right now that infrequently results in incomplete script/page loads. We are able to trap when this happens, and it can be solved by creating a new browser context, but there is no easy way to wholesale replace the original context of the test after all fixtures have run and page objects are initialized. The easiest solution is a simple retry, but we're finding that setting global or per-test retries to meet this need often causes tests to be re-run for other failures that are expected to (and indeed do) produce the same failing result. This is a waste of time and resources.
What we'd be after is something like
if (someSpecificCondition) {
test.retry({reason: 'Bad thing happened!', maxRetries: 2})
}
Any retries that may have already occurred from global or per-test configuration (such as is already readable via testInfo.retry) should count towards the limit specified as an argument to test.retry() - but the argument should be able to exceed the global or per-test configuration, which may be 0. I would expect retries triggered by test.retry() to increment testInfo.retry. I understand the maxRetries portion may present implementation challenges, but bounding is needed to prevent the possibility of an endless loop.
Please let me know if you have any questions or if there is a better way to meet this need. Thanks for your work on this amazing project and your consideration!
Along the same lines, we would find it useful to be able to skip retries under certain specific conditions.
For more context: We have enabled retries as part of our global configuration. We then have a fixture that checks for a certain condition as part of each test. If this condition is met, then we fail the test. In this specific case, we know that retries will not be useful and will only waste time.
A great addition to such feature would be a possibility to schedule retry until the end of current run. I am currently struggling with a flaky test where it's impossible to reproduce the problem locally, it seems like problem is related to k8s infra. Re-running problematic test at the end of everything could help in various cases. If failure is possibly related to unstable environment I would like to re-run that specific test later.
We would find this feature incredibly useful, and the enhancements proposed by nick-imvu and shwarcu would make it even better.
(Has anyone managed to implement this in the absence of official support?)
Coming here to also voice support for programmatical retry configuration.
In my case, I have a parametrized test suite that runs against multiple different servers. Some servers have unavoidable flakiness/conditions that require me to retry a subset of the test suite. I can only find out which server I am testing at runtime using an option that I pass in playwright.config.ts.
