headers_sent - behavior changed from PHPUnit 9 to 10

[DanielRuf]

Q	A
PHPUnit version	10.x
PHP version	8.1.13
Installation Method	PHAR

Summary

Starting with 10.0, tests with headers_sent() have a different behavior now.

Current behavior

The headers_sent() result differs from PHPUnit 9 to 10.

How to reproduce

<?php declare(strict_types=1);
use PHPUnit\Framework\TestCase;

final class BasicTest extends TestCase
{
	public function testEqual(): void
	{
		$string = 'a';

		if (!headers_sent()) {
			$string = 'b';
		}

		$this->assertSame($string, 'a');
	}
}

Expected behavior

The check for headers_sent() does not change from PHPUnit 9 to 10.

For reference, the results when testing with the original phar of PHPUnit 9.6.22 and 10.5.44:

php.exe .\phpunit-10.5.44.phar ./BasicTest.php
PHPUnit 10.5.44 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.1.13

F                                                                   1 / 1 (100%)

Time: 00:00.039, Memory: 26.00 MB

There was 1 failure:

1) BasicTest::testEqual
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'b'
+'a'

...\BasicTest.php:14

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

php.exe .\phpunit-9.6.22.phar ./BasicTest.php
PHPUnit 9.6.22 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 00:00.006, Memory: 26.00 MB

OK (1 test, 1 assertion)

I already tested and this affects every PHPUnit 10 version, including 10.0.0.

I checked the changelog and issues, so far I found no changes that could lead to this.

[DanielRuf]

By using the first parameter of headers_sent() I can see, that in PHPUnit 9 the headers were sent in phar://.../phpunit-9.6.22.phar/phpunit/Util/Printer.php, which is not the case in PHPUnit 10.

[DanielRuf]

What is the recommended solution to prevent that the tests fail but the application code still works? Maybe adding some check if the current SAPI is the CLI (similar to the following snippet)?

if (php_sapi_name() !== 'cli' && !headers_sent($file)) {
	$string = $file;
}

[staabm]

what are you trying to achieve by checking for headers_sent? it feels wrong to check for such a global side-effect and rely on it

[DanielRuf]

what are you trying to achieve by checking for headers_sent? it feels wrong to check for such a global side-effect and rely on it

The web application code is ~20 years old and depending on the result it adds a header or not. To me that does make sense. Or how would you check and prevent for "Headers already sent"?

[staabm]

based on the information given in this PR I can't say more then it feels wrong that a test relies on headers_sent, which is a assumption which can break pretty easily and its pretty hard to find the root cause why (fragile test).

I agree though, that PHPunit itself should "not send headers" (in case its technically possible) - but thats need to be investigated.

[sebastianbergmann]

it feels wrong that a test relies on headers_sent

It's common and not wrong (maybe not clean) in end-to-end tests. It used to work and I do not know why it no longer does. I would like to investigate that and ideally restore the previous behaviour.

[DanielRuf]

based on the information given in this PR I can't say more then it feels wrong that a test relies on headers_sent,

Not the test itself, the application code. My test in this issue is just a simple reproducible example. The real test does not directly use headers_sent but the application code used by it. At least in my case (and there our test asserts the result, which now differs due to this change).

[mvorisek]

headers_sent() is pointless in CLI and I would expect the default value on php startup unchanged.

Also note headers_sent() cannot be changed back once headers are sent.

[MrSuddenJoy]

headers_sent() is worthless as its outdated. There are newer, more modern ways of benchmarking webapps.

[staabm]

I had a look again into this.

the OPs test-result in PHPUnit 10+ makes sense to me. there is no header send anywhere, and therefore headers_sent() returns false.

The web application code is ~20 years old and depending on the result it adds a header or not. To me that does make sense. Or how would you check and prevent for "Headers already sent"?

sounds like the use-case you are having is not reflected in the code example you are providing.

@DanielRuf could you double check whether your reproducer really fits this description..? I would think there must be a header() call somewhere in the test, to make sense, right?

(I just try to figure out what the actual bug/problem is and how to reproduce, so I am not chasing ghosts. A proper reproducer is important, so we investigate the right thing)

[DanielRuf]

@staabm I'm not sure if people tried the reproducible examples from my initial comment using the PHAR for PHPUnit 9 and 10. Do these examples lead to different results for you?

[staabm]

I tried the examples and the behavior as of PHPUnit 10 makes more sense to me for the code given (even if it is not compatible with phpunit 9).

I don't see a bug yet

[DanielRuf]

Per https://github.com/sebastianbergmann/phpunit/issues/6121#issuecomment-2642287704

It's common and not wrong (maybe not clean) in end-to-end tests. It used to work and I do not know why it no longer does. I would like to investigate that and ideally restore the previous behaviour.

Ideally I would want to help, but currently have not much time. If I don't forget it after my holiday, I will try to do some more research via git-bisect and other ways.

Maybe we can find the root cause (and rationale behind this change) and make a decision, if this is actually a bug or desired behavior (and if there should be or is some flag to restore the previous behavior or not).

[staabm]

I don't say its wrong to use headers_sent() in tests, but the original example does not contain any code which sends headers, therefore I don't see why it should be expected to return true

[DanielRuf]

I don't say its wrong to use headers_sent() in tests, but the original example does not contain any code which sends headers, therefore I don't see why it should be expected to return true

Sure, It's in another part of the whole codebase. For reasons I can not upload whole controllers and more of our internal code.

I've opened the issue to understand where this change comes from. For this the reproducible example should be sufficient.

If it is a bug or feature (because maybe the previous behavior was wrong), is yet to determine. Personally I have not much to add, besides that I will try to find the time to dig deeper. And in the best case others will find the same issue and the final conclusion, which should be helpful.

[staabm]

For this the reproducible example should be sufficient.

no its not. the results for the given example look correct to me for PHPUnit 10+. its fine that PHPUnit 10.x produces a better result then PHPUnit 9.x for the given example, even if that looks like a BC break.

in case you don't find a better example which shows a real problem there is nothing we can do.