phpunit
phpunit copied to clipboard
sebastianbergmann
Support test execution order sorted by descending duration
We use PHPUnit with Paratest, allowing for parallel testrunners. This speeds up our tests on multi-core machines by a lot. One of the issues though, in terms of total test run time, is that some of our longer running tests don't get picked up until near the end. If these tests were executed first, it'd actually speed up the overall run time, since there would be other runners executing in parallel. As it is now, we might have 2 parallel runners executing long running tests at the very end, extending our total runtime by at least 30 seconds or more.
This is a request to add duration,reverse to the list of available order-by options.
My initial thought, and I cannot really explain why, is to not make reverse combinable with duration but rather introduce duration-ascending as an alias for duration and duration-descending for your use case.
I am curious whether @Slamdunk thinks this should be tackled in PHPUnit or in Paratest. After all, the case could be made that not only partitioning the tests but also sorting them for parallel execution should be the responsibility of the parallel test runner.
@sebastianbergmann that's a good point. Overall, paratest would be a more accurate place to handle this, since it can accurately distribute amongst the runners. Here is a comment on the subject: https://github.com/paratestphp/paratest/discussions/846#discussioncomment-9197441. It may be significantly more challenging though, since it's not currently making use of the result cache.
I am curious whether @Slamdunk thinks this should be tackled in PHPUnit or in Paratest. After all, the case could be made that not only partitioning the tests but also sorting them for parallel execution should be the responsibility of the parallel test runner.
@sebastianbergmann I think the opposite: sorting the tests has nothing to do with partitioning, as it has never been. Please implement the feature in PHPUnit and I'll ensure it works the same with ParaTest.
We are currently hindered to use the cache, but I'll open a different issue for that.
Please implement the feature in PHPUnit and I'll ensure it works the same with ParaTest.
Okay. PR welcome (for main), no promise when I might get to this myself.
@oojacoboo just to be clear, I'll work (someday) on https://github.com/sebastianbergmann/phpunit/issues/6078 but not on this very issue, which I consider up to you or anyone willing to step forward.
Full support of result cache operations, and thus all the current and future --order-by features, has been released in ParaTest v7.7.0
I think we see the OPs problem also in phpstan-src, where slow tests run pretty late and therefore we loose some time unnecessarily.
And I think we see it although we are using random ordering
We are using brianium/paratest ^6.5
