playwright-zephyr icon indicating copy to clipboard operation
playwright-zephyr copied to clipboard

feat(reporter): add `ignoreFailedRetries` option to avoid duplicate results during retries

Open yarikpavlin opened this issue 4 months ago • 2 comments

What

This MR introduces a new reporter option: ignoreFailedRetries.

When enabled, the reporter will skip all failed retry attempts if the test eventually passes. This helps reduce noise and avoids sending multiple results for the same test case.

Why

In Zephyr, each retry attempt can create a separate test result with a different status (e.g. "blocked", "failed") within the same test cycle. This leads to confusion and inflated reporting, even when the test eventually succeeds.

By enabling ignoreFailedRetries, only the first successful retry is reported — resulting in cleaner and more accurate test cycles.

Changes

  • ✅ Added ignoreFailedRetries flag to the reporter config (default: false)
  • ✅ Implemented filtering logic in onTestEnd
  • ✅ Updated README with usage example and rationale
  • ✅ Added test to verify ignoreFailedRetries behavior

Backward Compatibility

  • ✅ Feature is disabled by default, preserving current behavior for existing users

yarikpavlin avatar Aug 04 '25 23:08 yarikpavlin

@elaichenkov I'd appreciate your review whenever you have a chance 🙌 Let me know if anything needs to be adjusted!

yarikpavlin avatar Aug 04 '25 23:08 yarikpavlin

Hi @yarikpavlin,

Thank you for your contribution. Have you tested these changes with your zephyr? Which one do you have? Cloud or server? I don't have access to the zephyr anymore so can't test it.

elaichenkov avatar Aug 07 '25 02:08 elaichenkov