istio.io icon indicating copy to clipboard operation
istio.io copied to clipboard

TestCircuitBreaker is flaky

Open bison opened this issue 5 years ago • 3 comments

The TestCircuitBreaker test introduced in #6609 is flaky. It relies on examining the percentage of requests that succeed or fail to a service with circuit breaking configured, and ensuring the numbers fall within a given range. Unfortunately, the percentages tend to vary widely from run to run, so the test fails as often as it passes. The test was added at a time when this suite was not being run against pull requests regularly, so the failures went unnoticed for some time.

The test and corresponding documentation should be examined for any possible configuration changes that could make the results of the current method more consistent, or else the method of measurement in both the documentation and test should be changed.

Temporary fix: #7043

bison avatar Apr 14 '20 12:04 bison

This is pre #7043, but looking at the test results yesterday morning, for the last 19 tests run, there were 11 failures. In those 11 failures, 7 contained failures in TestTrafficShifting and 7 in TestCircuitBreaker (in some tests both failed).

ericvn avatar Apr 14 '20 14:04 ericvn

I think this is stale

howardjohn avatar Jun 04 '24 20:06 howardjohn

nope!

craigbox avatar Aug 06 '24 04:08 craigbox