RFE: kola should be able to run tests N times
kola run should have a --number / -n flag to run the test(s) up to n times and stop on the first failure. It would also be good to utilize the parallel flag so runnning a single test can be done faster, but even a serial implementation would be nice. Furthermore a --keep-going flag could be useful to always run n tests and report the success/failure rate.
I think that if an implementation of the -n flag is done we'd probably want to force the --keep-going functionality if the parallel flag is given. To properly implement the stop on first failure we'd have to plumb a lot of channels through the test clusters so when we do hit a failure we can initiate a clean exit/resource cleanup while also not reporting the status.