missing sent headers when 'RunInSeparateProcess' is used

[yurii-stickee-2023]

Q	A
PHPUnit version	11.5.3
PHP version	8.3.16
Installation Method	Composer / PHAR

Summary

Good day, @sebastianbergmann

Thank you for fixing this issue https://github.com/sebastianbergmann/phpunit/issues/6103 You have said there that it was executing for you once and it was just the print issue. The output was fixed there and that's great, but the problem with headers still exists for me.

I have very small understanding of PHPUnit internals, so I will try my best to explain the problem.

I am using Xdebug for testing headers (specifically xdebug_get_headers() ), and for some reason they are not properly sent (or not reported by xDebug) with RunInSeparateProcess while using PHPUnit 11.x (PHPUnit 10.x is working fine! Very strange situation, considering that it printed output twice as 11.x did)

Originally I thought that it somehow run test twice with headers and without, but after your output fix I am curious what is happening here now.

Current behavior

xdebug_get_headers() return empty, while sent headers are expected to be there.

My PHP setup is next

php -v
PHP 8.3.16 (cli) (built: Jan 19 2025 13:29:55) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.16, Copyright (c) Zend Technologies
    with Zend OPcache v8.3.16, Copyright (c), by Zend Technologies
    with Xdebug v3.4.0, Copyright (c) 2002-2024, by Derick Rethans

How to reproduce

0. Install xDebug and test that it works. Make some file like 1.php

<?php
header( "X-Test", "Testing" );
setcookie( "TestCookie", "test-value" );
var_dump( xdebug_get_headers() );

and run it as php 1.php. It should output

$ php 1.php 
/var/www/test-phpunit/1.php:4:
array(2) {
  [0] =>
  string(6) "X-Test"
  [1] =>
  string(33) "Set-Cookie: TestCookie=test-value"
}

I took example from https://xdebug.org/docs/all_functions#xdebug_get_headers

1. create composer project with "minimum-stability": "dev"
2. create test case tests/TestAnnotation4.php

<?php

#[\PHPUnit\Framework\Attributes\CoversNothing]
class TestAnnotation4 extends \PHPUnit\Framework\TestCase
{
    public static int $calls = 0;

    public function test_1()
    {
        $this->assertTrue(true);
    }

    #[\PHPUnit\Framework\Attributes\RunInSeparateProcess]
    public function test_case_2_check()
    {
        var_dump(++self::$calls);

        ob_start();
        header("X-Test", "Testing");
        echo 'asd';
        $content = ob_get_clean();

        $this->assertSame('asd', $content);
        $this->assertSame(['X-Test'], xdebug_get_headers());
    }
}

3. require PHPUnit as composer require phpunit/phpunit:11.5.3 -W
4. run test case as ./vendor/bin/phpunit tests/TestAnnotation4.php
5. See output with failed test

 ./vendor/bin/phpunit tests/TestAnnotation4.php
PHPUnit 11.5.3 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.16

.int(1)
F                                                                  2 / 2 (100%)int(1)


Time: 00:00.059, Memory: 8.00 MB

There was 1 failure:

1) TestAnnotation4::test_case_2_check
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
-Array &0 [
-    0 => 'X-Test',
-]
+Array &0 []

/var/www/test-phpunit/tests/TestAnnotation4.php:26

FAILURES!
Tests: 2, Assertions: 3, Failures: 1, PHPUnit Deprecations: 2.

NOTE! If you run it with the fix from https://github.com/sebastianbergmann/phpunit/issues/6103 you will see that output is fixed (it outputs int(1) once), but the issue with headers is still there.

composer require phpunit/phpunit:11.5.x-dev -W

 ./vendor/bin/phpunit tests/TestAnnotation4.php

PHPUnit 11.5.3 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.16

.int(1)
F                                                                  2 / 2 (100%)

Time: 00:00.059, Memory: 8.00 MB

There was 1 failure:

1) TestAnnotation4::test_case_2_check
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
-Array &0 [
-    0 => 'X-Test',
-]
+Array &0 []

/var/www/test-phpunit/tests/TestAnnotation4.php:26

FAILURES!
Tests: 2, Assertions: 3, Failures: 1, PHPUnit Deprecations: 2.

Expected behavior

The expected behavior is that xdebug_get_headers() returns sent headers and not empty array.
For example, in PHPUnit 10.x it works as expected and returns sent headers

composer require phpunit/phpunit:10.5 -W

./vendor/bin/phpunit tests/TestAnnotation4.php

PHPUnit 10.5.0 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.16

./var/www/test-phpunit/tests/TestAnnotation4.php:18:
int(1)
.                                                                  2 / 2 (100%)/var/www/test-phpunit/tests/TestAnnotation4.php:18:
int(1)


Time: 00:00.059, Memory: 8.00 MB

OK (2 tests, 3 assertions)

that is what I expected before. It works correctly in PHPUnit 9.x with annotations too.

Best regards

[yurii-stickee-2023]

If you do not mind I connect these issues as they may be related https://github.com/sebastianbergmann/phpunit/issues/6121

[aeytom]

Can confirm:

• tests working in phpunit10, php8.2
• not working in phpunit12, php8.3 with process isolation enabled

[sebastianbergmann]

Comparing "PHPUnit 10 + PHP 8.2" to "PHPUnit 12 + PHP 8.3" is not helpful, sorry.

[aeytom]

I can try to debug this. But I need some relevant code entry points for the isolated processes.

[aeytom]

isolated tests working in PHP 8.3.17, phpunit 12.0.5 when

xdebug.mode=gcstats[,coverage]

or with listening xdebug

xdebug.mode=debug[,coverage]
xdebug.start_with_request=yes

[mttsch]

@aeytom @sebastianbergmann I ran into the same or at least a similar problem where xdebug.mode was, if I remember correctly, set to only coverage to create a coverage report (but did not include debug) and xdebug.start_with_request was set to trigger in an ini file.

The relevant test using xdebug_get_headers() and #[RunInSeparateProcess] did work when running PHPUnit with coverage, but failed without coverage. The test (without coverage) worked with PHPUnit 9 and 10, but failed with PHPUnit 11 and 12 (each time the latest minor/patch release), all while keeping the same PHP version 8.4.x.

In my case, I could point it down to https://github.com/sebastianbergmann/phpunit/commit/4e52bb6#diff-42df2a075f4f6210e6960e016357a9eddc7007ac25766f34ca0679c3e4d70d55R194 that, if I understand it correctly, will disable Xdebug completely for the child process if in the parent process Xdebug debugger was not started - and it was not started because it is not included in xdebug.mode, see above. Basically, this line overwrote my own xdebug.mode setting. Commenting out this line resulted in the test working again with PHPUnit 12, with and without coverage mode in PHPUnit.

I worked around it by changing my Xdebug ini setting.

[sebastianbergmann]

Interesting, thanks.

@staabm Can you have another look at https://github.com/sebastianbergmann/phpunit/commit/4e52bb6#diff-42df2a075f4f6210e6960e016357a9eddc7007ac25766f34ca0679c3e4d70d55R194? Thanks!

[staabm]

@sebastianbergmann I was able to reproduce the issue and can also see how it depends on https://github.com/sebastianbergmann/phpunit/commit/4e52bb6#diff-42df2a075f4f6210e6960e016357a9eddc7007ac25766f34ca0679c3e4d70d55R194

I could look into, whether we can stop fiddling with xdebug mode, when a process-isolated test declares #[RequiresPhpExtension('xdebug')]. how does that sound?

[mttsch]

@staabm In my case, the test was annotated to require function xdebug_get_headers but #[RequiresPhpExtension('xdebug')] would achieve the same, I guess 👍

[staabm]

Hmm... Actually I think it would be best if the test also would declare its requirement on a certain xdebug.mode ini value.

Having a test depend on a global php.ini setting without a proper check sounds like asking for trouble

[mttsch]

I get your point.

For this specific issue, I would have to investigate what the relevant ini setting would be for xdebug_get_headers() to return the headers. Potentially xdebug.mode != off, thus it would be more of a negative check against an ini setting?

PS: I only just realized that RequiresSetting might be for ini settings, I searched https://docs.phpunit.de/en/10.5/attributes.html for "ini" in the past but did not find any results.

[staabm]

PS: I only just realized that RequiresSetting might be for ini settings, I searched docs.phpunit.de/en/10.5/attributes.html for "ini" in the past but did not find any results.

yeah, I have similar feelings. it would be more explicit if it would be named RequiresIniSetting

[staabm]

I could look into, whether we can stop fiddling with xdebug mode, when a process-isolated test declares #[RequiresPhpExtension('xdebug')]. how does that sound?

implemented in https://github.com/sebastianbergmann/phpunit/pull/6353

[staabm]

can be closed, as the fix was merged