sebastianbergmann
Tests for test suite reordering fail
I pulled #6338 into 10.5 and merged it from there to 11.5, 12.4, 12.5, and main.
On PHPUnit >= 12, two tests, order-by-defects-test-classes-with-defects.phpt and order-by-defects-test-methods-with-defects.phpt, now fail.
These test failures need to be investigated. Hopefully, these tests fail for the obvious reason: that the result cache fixture file needs to be updated.
@staabm I can't see the forest for the trees at the moment, maybe a fresh pair of eyes is needed? Could you take a look? Thanks!
Looking at tests/end-to-end/execution-order/order-by-defects-test-classes-with-defects.phpt, I wonder why the expectation is that after the 2 failled tests namely testThree and testTwo - the 2 succeeding tests are expected in the order testFour, testOne.
shouldn't they be executed in the order of declaration in the test-class, which means testOne before testFour (and thats what the test runner in PHPUnit 12.x is doing)
I guess what I am saying is, that the tests looks for me to work like expected starting with 12.x. maybe I miss something?
Thank you for your feedback, I did not think of this possibility.
As explained in https://github.com/sebastianbergmann/phpunit/pull/6338#issue-3359042338, to me, these new tests are characterization tests meaning they characterize the current behaviour. If you are correct, then the characterization tests I originally wrote for PHPUnit 10 now show us that PHPUnit 12 behaves differently.
One PR I remember related to sorting is https://github.com/sebastianbergmann/phpunit/pull/6263
which landed in PHPUnit 11.5+, which means at least for cases which use data-provider there might be a difference between PHPUnit 10 and PHPUnit 11+ (but this does not explain the difference we see regarding the above comments)
The new tests work as-is for both PHPUnit 10 and PHPUnit 11, so the PR you mention is unrelated.