Extension point for single test execution

[lstrojny]

protected function runTest() in PHPUnit\Framework\TestCase provided an extension point to change the actual run implementation. That enabled various use-cases around exception handling and specialized test execution.

One additional use-case that has not been mentioned yet is to retry tests that might be unstable.

It would be terrific if PHPUnit could provide an officially blessed extension point to change test execution.

See related issues:

• https://github.com/sebastianbergmann/phpunit/issues/6172
• #5900
• https://github.com/sebastianbergmann/phpunit/issues/5630

Here is an example of how runTest was used for a retry implementation:

/**
 * @mixin TestCase
 * @phpstan-require-extends TestCase
 */
trait ResilientTest
{
    #[Override]
    final protected function runTest(): mixed
    {
        if (!self::shouldRetryTest($this)) {
            return parent::runTest();
        }

        $maxRetries = self::retryTimes($this);
        for ($retry = 0; $retry < $maxRetries; ++$retry) {
            try {
                return parent::runTest();
            } catch (SkippedTest|IncompleteTest $e) {
                throw $e;
            } catch (Throwable $e) {
                if ($retry === $maxRetries - 1) {
                    throw $e;
                }
                error_log(sprintf('Retrying %s (%d of %d)', $this->name(), $retry + 1, $maxRetries));
            }
        }

        return parent::runTest();
    }

    private static function shouldRetryTest(TestCase $testCase): bool
    {
        return self::retryTimes($testCase) > 1;
    }

    private static function retryTimes(TestCase $testCase): int
    {
        $class = new ReflectionObject($testCase);

        $currentClass = $class;
        do {
            $classAttributes = $currentClass->getAttributes(Retry::class);
            if (count($classAttributes) > 0) {
                return $classAttributes[0]->newInstance()->times ?? 1;
            }

            $methodAttributes = $currentClass->getMethod($testCase->name())
                ->getAttributes(Retry::class);
            if (count($methodAttributes) > 0) {
                return $methodAttributes[0]->newInstance()->times ?? 1;
            }
        } while (($currentClass = $currentClass->getParentClass()) !== false);

        return 1;
    }
}

[sebastianbergmann]

That enabled various use-cases around exception handling and specialized test execution.

I am not sure what you mean with "exception handling" here, but TestCase::registerFailureType() can be used to register additional exception types that should be treated as an assertion failure instead of as an error.

This is an example of our approach with TestCase::run(), TestCase::runBase(), and TestCase::runTest(). These methods are either declared final and marked as internal and therefore not covered by the backward compatibility promise for PHPUnit or they are declared private. It was possible to overwrite them in the past, which caused all sorts of problems. Instead, and where it is sensible, we provide methods such as the aforementioned TestCase::registerFailureType() for customizing very specific aspects of running a test.

It would be terrific if PHPUnit could provide an officially blessed extension point to change test execution.

I do not see how this would be possible without causing more problems than this would be worth.

One additional use-case that has not been mentioned yet is to retry tests that might be unstable.

This feature is discussed in #6182.