Option to retry failed commands

[jheld]

If an environment fails the test run, I think it would be nice to have support for automatically retrying the test environment.

Brute force would be simply to retry (with a parameter for the number of max retries). I'd also be interested in a way to tell the environment to run an altered command on the retry, in case only a subset of tests for that env would need to be re-run.

I think this probably would live as a plugin, but I'm not sure if the tox plugin system would even be able to support this sort of flow change.

I hope to research this more and if I find something before anyone responds, I'll post back.

With the hook system there is: tox_runtest. I'm uncertain exactly how the order of external registered plugins will be added against the one defined by tox. I am assuming they go first (esp given it runs with completion on first hook returning non None).

For a retry plugin to work, I think it would have to use the hookwrapper option, and as such it will call the other hooks until they succeed or max-retries reached. Still I'm a little concerned about how the order of plugin hooks is determined, but I hope that won't stop me from trying to do this!

[ssbarnea]

As I faced a related issue where a specific command I run requires network access and may randomly fail. It would have being cool to be able to retry it by using a prefix, similar to how the - works to ignore the result.

Maybe a "?" prefix could mean repeat it 3 times if exit code is not zero (maybe configurable).

[gaborbernat]

I don't want to overload the command line even further to make it more cryptic. I'm inclined to add some command_retry option though that would allow users to specify the retry counts per command and then zip the two arrays for command. A PR for that would be welcomed.

[mpatnode]

+1 (Not seeing the upvote button.)