Abstract *Test classes cannot be found

[derrabus]

Q	A
PHPUnit version	10.0-dev
PHP version	8.1.6
Installation Method	Composer

Summary

I wanted to try out PHPUnit 10-dev on a project that has a green and (almost) deprecation-free test suite running on PHPUnit 9.5. In that project, there are several abstract classes following the naming convention AbstractSomethingTest. Then there are non-abstract child classes which are actually executable test cases.

When running PHPUnit 10, it crashes on the first of those abstract classes, claiming that it could not find the class:

PHP Fatal error:  Uncaught PHPUnit\Runner\Exception: Class 'MyApp\Tests\SomeFeature\AbstractSomethingTest' could not be found in '/path/to/project/tests/SomeFeature/AbstractSomethingTest.php'. in /path/to/project/vendor/phpunit/phpunit/src/Runner/TestSuiteLoader.php:147
Stack trace:
#0 /path/to/project/vendor/phpunit/phpunit/src/Runner/TestSuiteLoader.php(125): PHPUnit\Runner\TestSuiteLoader->exceptionFor('MyApp\\Tests\\...', '/path/to/projec...')
#1 /path/to/project/vendor/phpunit/phpunit/src/Framework/TestSuite.php(295): PHPUnit\Runner\TestSuiteLoader->load('/path/to/projec...')
#2 /path/to/project/vendor/phpunit/phpunit/src/Framework/TestSuite.php(307): PHPUnit\Framework\TestSuite->addTestFile('/path/to/projec...')
#3 /path/to/project/vendor/phpunit/phpunit/src/TextUI/Configuration/Xml/TestSuiteMapper.php(74): PHPUnit\Framework\TestSuite->addTestFiles(Array)
#4 /path/to/project/vendor/phpunit/phpunit/src/TextUI/Configuration/TestSuiteBuilder.php(56): PHPUnit\TextUI\XmlConfiguration\TestSuiteMapper->map(Object(PHPUnit\TextUI\XmlConfiguration\TestSuiteCollection), '', '')
#5 /path/to/project/vendor/phpunit/phpunit/src/TextUI/Application.php(243): PHPUnit\TextUI\Configuration\TestSuiteBuilder->build(Object(PHPUnit\TextUI\CliArguments\Configuration), Object(PHPUnit\TextUI\XmlConfiguration\LoadedFromFileConfiguration))
#6 /path/to/project/vendor/phpunit/phpunit/src/TextUI/Application.php(95): PHPUnit\TextUI\Application->handleArguments(Array)
#7 /path/to/project/vendor/phpunit/phpunit/src/TextUI/Application.php(78): PHPUnit\TextUI\Application->run(Array, true)
#8 /path/to/project/vendor/phpunit/phpunit/phpunit(90): PHPUnit\TextUI\Application::main()
#9 {main}

Stacktrace redacted to remove references to the actual project. I could provide a minimal reproducer if necessary.

This error message is pretty odd because the class PHPUnit complains about is defined exactly in the file mentioned in the exception message.

If I rename that one class from AbstractSomethingTest to SomethingTestCase and run PHPUnit again, it crashes on the next class following that pattern. If I rename all of the affected classes, PHPUnit runs fine.

Apparently, PHPUnit does not like abstract classes anymore that use the Test suffix. I wasn't sure if this was an intended breaking change or not, but I could not find documentation on this. Renaming those classes wouldn't be a big deal in my case (it would only affect 10 or so classes). But if this is really an intentional change, I believe something must be done about the error message at least because that message is highly misleading.

[sebastianbergmann]

Apparently, PHPUnit does not like abstract classes anymore that use the Test suffix.

Correct.

I believe something must be done about the error message at least because that message is highly misleading.

Agreed.

[IonBazan]

I think this should produce a deprecation warning in 9.5 branch so that users are informed about this change in advance. I tried to add it in https://github.com/sebastianbergmann/phpunit/blob/a4246d5352b16c87547505d1af713cf348e3646f/src/Framework/TestSuite.php#L457 but it produced many false-positives as this part is executed for each child class too.

[derrabus]

Thank you for fixing this issue!

[kevinpapst]

Any chance you @sebastianbergmann can share why that change was introduced?

I assume there are many projects following this obvious inheritance and naming scheme. I find it weird that, after years of compatibility, the test framework now forces me to rename them.

Would be "less painful" 😁 if we would understand what the technical reason for that change was. The original issue #5132 does not share any insights.

[k00ni]

One note: If you have files (with Test in the name) inside the tests folder, which do non-PHPUnit related testing, it will also complain with a warning.