`#[Retry]` attribute to support retrying flaky test

[santysisi]

✨ Add Retry Attribute for Retrying Flaky Tests

This PR introduces a new Retry attribute that can be applied to individual test methods in PHPUnit. Its goal is to help deal with flaky or unstable tests that occasionally fail due to non-deterministic reasons (e.g. network issues, race conditions, external dependencies).

🔧 How it works

The Retry attribute allows a test to be automatically retried a specified number of times before being marked as failed. You can also control the delay between retries, and optionally restrict retries to specific exception types.

🧪 Usage

Here’s how you can use it:

#[Retry(3)] // Retry up to 3 times on any exception
public function testFlakyApi(): void
{
    // test logic
}

You can also specify a delay (in seconds) between retries:

#[Retry(2, delay: 2)] // Retry twice, with 2 seconds delay between attempts

And restrict retries to certain exception types only:

#[Retry(4, retryOn: TimeoutException::class)]

All parameters:

• maxRetries (int) — how many times the test can be retried.
• delay (?int) — seconds to wait before retrying. Default: 0
• retryOn (?string) — exception type to retry on. If omitted, retries on any throwable.

✅ Example

#[Retry(2)]
public function testRetriesUntilMaxAttemptsThenSucceeds(): void
{
    static $attempt = 0;

    if ($attempt++ < 2) {
        throw new RuntimeException('Transient failure');
    }

    $this->assertSame(3, $attempt);
}

📈 Benefits

• Stabilize CI pipelines by automatically retrying flaky tests
• Fine-grained control over retry behavior per test
• Non-intrusive: no need to change existing test logic
• Optional support for retrying only on specific exceptions

🔄 Prior art

Other popular testing frameworks already provide similar functionality:

• NUnit: Retry Attribute
• JUnit Pioneer: RetryingTest

[santysisi]

This PR could also offer a complementary approach to the discussion in #5718: Bring back --repeat CLI option.

While --repeat focuses on re-running tests via the CLI, the #[Retry] attribute is designed for more fine-grained control, enabling retries at the test method level, optionally with delay or exception-based filtering. It might be a useful alternative for handling flaky tests directly in code, without requiring changes to CI configs or CLI commands.

Thanks again, looking forward to your feedback! 😊

[nikophil]

Hey @santysisi

Thanks for this PR! Any chance that you add the CLI option along with the attribute?

[sebastianbergmann]

Any chance that you add the CLI option along with the attribute?

I do not see how a CLI option would be relevant in the context of what this PR is about.

[sebastianbergmann]

I think that the functionality you propose here is useful.

However, the current implementation can "only" be considered an early proof of concept in my opinion. Retried tests are not run on a fresh instance of the test case object, the before-test, after-test, etc. template methods are not run for retried tests, etc. Furthermore, no events are emitted to record the fact that a test was retried. At least as far as I can see.

While I do think this would be a nice addition, I will not work on a "real" implementation myself. I am open to a more complete PR, of course.

[santysisi]

Hi @sebastianbergmann , thanks for your comment :smile:. I really appreciate it!

I'd like to give it another try, now taking the new requirements into account. Please let me know if I’m on the right track:

• Fresh Test Case Instances for Retries, allowing the proper execution of lifecycle/template methods (setUp, tearDown, etc.).
• Emitting Events for Retried Tests, so retries can be logged or tracked appropriately.

Does that align with what you had in mind?

Also, just to confirm, do you agree with the behavior described here

[santysisi]

Hi @sebastianbergmann , thanks for your comment 😄. I really appreciate it!

I'd like to give it another try, now taking the new requirements into account. Please let me know if I’m on the right track:

• Fresh Test Case Instances for Retries, allowing the proper execution of lifecycle/template methods (setUp, tearDown, etc.).
• Emitting Events for Retried Tests, so retries can be logged or tracked appropriately.

Does that align with what you had in mind?

Also, just to confirm, do you agree with the behavior described here

Hi @sebastianbergmann, I hope you're doing well 🙂 Could you please confirm if this aligns with what you had in mind, or if I might be going in the wrong direction? 🙏 Thanks in advance! 😊

[sebastianbergmann]

Could you please confirm if this aligns with what you had in mind, or if I might be going in the wrong direction? 🙏

I am sorry that I have not gotten around to look into this yet.