executionOrder setting is not properly documented

[Bilge]

First, the only mention of a feature governing the execution order of tests is in an appendix page, which provides poor exposure to a major feature.

Next, the only thing the docs have to say about the executionOrder flag is that, This attribute configures the order in which tests are executed. This is clearly not enough to explain a feature that currently has no less than eight different configuration values and accepts multiple values thereof.

Specifically, as a user I want to know:

• What does defects do?
• What does depends/no-depends do?
• Assuming duration orders tests by duration, how does PHPUnit know how long my tests take, to produce that ordering?
• I know from discussion with other developers that there is more than meets the eye to random, specifically that seed values can be provided. If true, this must be documented.
• What does size do? How does PHPUnit know the "size"?
• In which version was this feature, and the respective settings, added?

[epdenouden]

Hello @Bilge!

Thank you for taking the time to ~detail what documentation is missing~ give me a long overdue spanking. No Nikolaus presents for me this year!

I will document in the official manual:

• what console flags and configuration settings do what
• which functionality is available in which version of PHPUnit. Since its introduction in version 7 there have been significant changes along the way
• when a test is considered 'defective' and when it is not, as this can be influenced by other options
• what is a dependency as seen through the eyes of the @depends annotation, as this isn't exactly what is colloquially considered a 'software dependency'
• the random+seed option is very simple and useful! A missed opportunity by me to invite more people play with that option.

As I work on PHPUnit in my off hours it will take me a bit of time, so in the meantime here are links to original pull requests with examples:

• first execution reordering features which brought --order-by with the modes reverse and random order, plus a basic @depends enforcer (within TestCase-scope)
• reordering based on results of previous run which introduced the defects option

I hope the features have been useful to you in the meantime. If you have any further suggestions for improving the user experience don't hesitate to add those here.

[Bilge]

Whilst it may be the case that spanks are due, it should be clear that such spanks were not directed at anyone in particular. Nevertheless, I am very grateful you have taken responsibility to receive them and I'm sure you will do a great job improving the documentation! :clap:

[clement-fleury-assoconnect]

🆙 on this issue ;)

[AllenJB]

The documentation indicates that combinations are possible, using ',' as a separator (without specifying limitations), but when you choose certain (or rather, other than a certain list) of combinations, you get a warning that this is not valid, but no reasoning as to why.

In my case I was trying to use "defects,random" (using PHPUnit 9.1.1), which I take to mean "run defective tests from last run first, then run everything else in random order".

[amcsi]

I was trying to change how tests were sorted when I noticed that it didn't seem to work. I found this issue, but this seemed to be about the docs. Although what each order flag does is indeed not well documented, I believe that regardless, there's a bug that makes them not work in any way. Please see the issue I've opened: sebastianbergmann/phpunit#4619

[Bilge]

Any year now...

[localheinz]

Fixed with #308.