PHPUnit tests bug out when a --coverage option is used.

[hopeseekr]

The full bug report can be found at https://github.com/sebastianbergmann/phpunit/issues/4459.

This issue is to make you guys aware of it.

I mainly submitted this report because I hadn't seen this error in over 50,000+ PHP projects we've been able to run PHPUnit on to date.

Q	A
PHPUnit version	8.5.8 and 9.3.10 [latest]
PHP version	7.2.33, 7.3.22, and 7.4.10 [latest]
Installation Method	composer

The Problem

As part of my Bettergist Collective project, I have been running phpunit (and a host of other utilities) against (eventually) the entire catalog of PHP packages in the Composer / Packagist.org universe. I currently maintain the PHP Packagist Archive, which already contains quarterly backups of every single package.

As such, I have run thousands and thousands of phpunits across tens of thousands of projects (currently ~50,000+), which (tell @sebastianbergmann !) is probably a better test suite of PHPUnit itself over anything on the planet (even more than what they must have!). I even test every package against PHPUnit 7.2 through 8.0 (if composer manages to install for each) ;_)

This is a new one for me.

I have never seen this in 23 years as a PHP developer and 20 year-user of PHPUnit, so that is why I'm reporting it.

Current behavior

Tests run normally without any flags.

However, when any of the --coverage options is used, phpunit will report basically every test as "Risky".

How to reproduce

I discovered this when I got to the hs of PHP projects.

phpunit --coverage-text

It may very well have to do with arcane settings in their phpunit.xml.dist:

         beStrictAboutChangesToGlobalState="true"
         beStrictAboutOutputDuringTests="true"
         beStrictAboutResourceUsageDuringSmallTests="true"
         beStrictAboutCoversAnnotation="true"

I have never seen any of those out in the wild before...

Expected behavior

I would expect PHPUnit to run in a manner where tests are not marked as Risky, or otherwise fail, just because a --coverage option was used.

Thank you!

I routinely use box to package phars, particularly for utiltiies used in my Bettergist Collective.

Cheers!

[owenvoke]

I think I understand what Sebastian is getting at (and it looks like PHPUnit is correct). 🤔 Basically the test in AnnotationDumperTest is only marked as covering \KevinGH\Box\Annotation\AnnotationDumper, however on line 50 it calls the DocblockParser::parse() method. Hence PHPUnit is throwing an error because it's being strict about the coverage as expected from the config. 👍🏻

Looks like the following diff can be applied to resolve this. 🤷🏻 However, this seems to apply in quite a few of the other tests. 🤔

diff --git a/tests/Annotation/AnnotationDumperTest.php b/tests/Annotation/AnnotationDumperTest.php
index 5772ff8..5b74f80 100644
--- a/tests/Annotation/AnnotationDumperTest.php
+++ b/tests/Annotation/AnnotationDumperTest.php
@@ -19,6 +19,7 @@ use PHPUnit\Framework\TestCase;

 /**
  * @covers \KevinGH\Box\Annotation\AnnotationDumper
+ * @covers \KevinGH\Box\Annotation\DocblockParser
  */
 class AnnotationDumperTest extends TestCase
 {

beStrictAboutCoversAnnotation checks code that is run by a test using a @covers annotation, and will mark the test as risky because it isn't marked as covering that code.

[theofidry]

Maybe beStrictAboutCoversAnnotation should not be used then. My usage of @covers in Box is mostly to tell what is tested, I do not want to have to provide the exhaustive list of what it uses on top of that

[owenvoke]

@theofidry, yeah. That probably makes more sense. 👍🏻

[villfa]

@hopeseekr the problem should be fix now. May you confirm it?

[jrfnl]

This looks to have been fixed. Should this issue be closed ?