playwright
playwright copied to clipboard
microsoft
[Feature] Add trace option for 'on-first-retry-failure'
Let us know what functionality you'd like to see in Playwright and what your use case is. Do you think others might benefit from this as well?
It'd be nice to have a trace option that a mixture of retain-on-failure and on-first-retry. A possible flag for on-first-retry-failure' would only make traces for those that fail on the first retry and doesn't retain those that pass on the second retry (flaky test). This way it's easy to determine traces for those that are true failures rather than mixing in the traces that are actually passing on the second try. I think this is a feature that others would end up using as well.
Marking as P3 collecting feedback. So essentially, you want us to erase the trace if on-first-retry is set and the test passed on retry? We assume that you are using some kind of reporter (html reporter?) that lists failed tests and makes it is easy to follow the link to open its trace. We did not think you would be looking at the file system and opening all the traces.
We are running our tests in a CI environment and are publishing the test results folder after the build. We want to be able to erase any traces that ended up passing so that it is easier to navigate and find traces that represent actual issues when we view the uploaded artifacts. We are using the JUnit reporter as of now.
logically to have records about the first fail, not about the first retry fail?
logically to have records about the first fail, not about the first retry fail?
The idea would be to retain all traces of those that are true failures, rather than flaky tests (which I would like to be able to handle separately)
You run the risk of a flaky test failing twice, creating a false positive. And because you're assuming that it's not flaky, it will make it more confusing to resolve.
I recommend just put some elbow grease into fixing the flaky tests. Run them at full speed locally, and use slowmo:1000 to slow them down and expose any timing flaws that way. Worked well for me, no flaky tests anymore!
We would also like this feature.
We are using the on-first-retry option, but it makes no sense for us to save the trace if test passed on retry.
It just makes the report bigger.
This would help to keep the trace for failed test only, if retry is successful no need to have the trace. retain-on-failure could be an option here, but it duplicates the trace in case you have a retry, keeping the trace for first execution + retry.
