ginkgo
ginkgo copied to clipboard
onsi
Allow flakedetection by running tests multiple times and requiring all runs to pass
Hey there,
I noticed ginkgo does not seem to have anything like go tests -count flag which will run all tests count times and require all runs to be a success in order to consider the test to be a success. This is extremely useful for flake detection. Would you consider a PR that adds that?
This is kind of the inverse of https://github.com/onsi/ginkgo/issues/260 which had as goal to tolerate flakes, this has as goal to always run tests multiple times and never tolerate flakes.
Ginkgo does have -untilItFails which reruns your test forever until a failure is found. I like the idea of something like -count and am a fan of finding and weeding out flakes over tolerating flakes.
I'm in the middle of working on v2 and hope to push something out publicly in the next couple of weeks. I'd prefer to see this PR on that branch after I make that first public push. Would that work for you?
Ginkgo does have -untilItFails which reruns your test forever until a failure is found
Nice, that helps at least locally but unfortunately not for CI.
I'm in the middle of working on v2 and hope to push something out publicly in the next couple of weeks. I'd prefer to see this PR on that branch after I make that first public push. Would that work for you?
Hm, I think until v2 actually ends up in downstream projects it will take quite a bit of time, do you think it would be difficult or hard to maintain to add this to both?
not hard. just adds to the tech debt i need to maintain. i'm hoping v2 gets adopted pretty quickly given there are relatively few breaking changes.
go ahead and PR it in. i'll port it over to v2 when it's time.
also - i'm thinking this would go in the cli alongside untilItFails. does that make sense?
If we were designing this from the start, it would have made sense to make -untilItFails a flag that takes an optional limit. I've not looked at the code, but it could be nice if we could move in that direction. Maybe introducing a new runXTimes (with a better name) flag that's an alias of untilItFails. Please feel free to completely ignore this suggestion.
Putting it into the cli means that ppl have to install the cli to use it, I would prefer it if that wasn't needed and this could just be passed as an arg to go test
I'd prefer to see this in the cli where this kind of functionality is currently implemented. And I'd like to push more people towards the cli over go test as it implements ginkgo's parallelization support.