`--filter FilterWhichDoesNotMatchAnything` should exit with non-zero exit code

[staabm]

Q	A
PHPUnit version	a62124f38
PHP version	8.3.23
Installation Method	Composer / PHAR

How to reproduce

➜  phpunit git:(main) php phpunit --filter FilterWhichDoesNotMatchAnything          
PHPUnit 12.3-ga62124f38 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.23
Configuration: /Users/m.staab/Documents/GitHub/phpunit/phpunit.xml

No tests executed!
➜  phpunit git:(main) echo $?            
0

➜  phpunit git:(main) php -v                                              
PHP 8.3.23 (cli) (built: Jul  1 2025 16:52:12) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.23, Copyright (c) Zend Technologies

Expected behavior

I think in case we do not run any tests and had a --filter defined, PHPUnit should return a non-zero exit code

[staabm]

I think the same should be true for --testsuite and --group based filters:

➜  phpunit git:(main) php phpunit --group test-abc
PHPUnit 12.3-ga62124f38 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.23
Configuration: /Users/m.staab/Documents/GitHub/phpunit/phpunit.xml

No tests executed!
➜  phpunit git:(main) echo $?
0
➜  phpunit git:(main) php phpunit --testsuite test-abc
PHPUnit 12.3-ga62124f38 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.23
Configuration: /Users/m.staab/Documents/GitHub/phpunit/phpunit.xml

No tests executed!
➜  phpunit git:(main) echo $?                         
0

[kubawerlos]

What about running without any parameters and still getting "No tests executed!" (because of no tests)?

What about a simple relation, "No tests executed!" -> "a non-zero exit code"?

[staabm]

this might work, too.

I tried to talk about my actual use-cases. but it might work for a more general case.

not sure about possible BC impact though.

[joteejotee]

I'd like to work on this issue if nobody else is working on it.

[staabm]

We have no signal from Sebastian yet whether a fix would be accepted (he is on vacation atm).

But if you want to work on it - go ahead.

[joteejotee]

That makes sense—since we haven’t heard Sebastian’s opinion yet, I’ll hold off for now. Thanks for letting me know!

[stof]

There is already a --fail-on-empty-test-suite config setting (the XML config file also supports it). Isn't it solving that need already ?

[staabm]

Isn't it solving that need already ?

thanks, wasn't aware of --fail-on-empty-test-suite.

I think the behaviour of --fail-on-empty-test-suite should be the default for --filter (or maybe even in general).

[stof]

@staabm just set failOnEmptyTestSuite="true" in your XML config file (or failOnAllIssues="true").

Right now, none of the failOn* setting defaults to true, so this is consistent.

[staabm]

thanks for the examples. that indeed proves that my initial problem is actually not the same as the feature --fail-on-empty-test-suite seems to provide.

I think its a DX problem that PHPUnit reports a "success" result when its instructed to run with a certain filter but no tests match such filter. it would be useful for any tool (KI Agent, CI Job, batch file,..) invoking phpunit that no tests were run (by default, without an additional required config flag)