exitfirst should mark all unrun tests as "failed" for the purpose of last-failed and friends

[xmo-odoo]

--exitfirst and --last-failed work fine with xdist. Sadly stepwise doesn't (#458).

One might think that stepwise could be emulated by running with both exitfirst and last-failed, however it looks like exitfirst does not consider any of the yet-to-run tests "failed", as a result last-failed will run only the test which actually failed, instead of resuming from where it stopped.

[RonnyPfannschmidt]

Step wise needs a explicit xdist integration, reporting wrong Data to shoehorn it into running will just create more trouble

[xmo-odoo]

I get that, but honestly even independent of that issue I always think it's weird that last-failed simply doesn't see the tests which have yet to run... unless there are no failures at all. I think it would make sense for existfirst and lastfailed to use (and set) whatever mechanism stepwise uses for "stuff still to run".

[RonnyPfannschmidt]

I regularly use the combination of lastfailed and exit first to iteratively cull broken tests after internally breaking changes,

The suggestion would break that

[xmo-odoo]

I regularly use the combination of lastfailed and exit first to iteratively cull broken tests after internally breaking changes,

When I run that combination, pytest stops at the first failure then only reruns that specific failure and nothing else (which is exactly my issue with it), is that really the intent? Because to me that's mostly a worse stepwise.

[RonnyPfannschmidt]

Yes, that is the intent, the goal being to chip away one failing test at a time until it does the next more complete rerun