old_issues_repo icon indicating copy to clipboard operation
old_issues_repo copied to clipboard

Circuit breaker doesn't work for httpConsecutiveErrors

Open jeson114 opened this issue 6 years ago • 4 comments

Y Is this a BUG or FEATURE REQUEST?: BUG Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?: Y Bug: Y/N Y What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version : Version: 0.7.1
kubectl version : Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Is Istio Auth enabled or not ? Did you install the stable istio.yaml, istio-auth.yaml.... or if using the Helm chart please provide full command line input. Y What happened: Circuit breaker doesn't trip on httpConsecutiveErrors: 1 (for 500 response). All requests pass through and give a 500 instead What you expected to happen: Circuit breaker should trip and should return 503(Service Unavailable). Inst How to reproduce it: Follow the steps Circuit breaker setup Once httpbin is up you can simulate 500 with it Request : kubectl exec -it $FORTIO_POD -c fortio /usr/local/bin/fortio -- load -c 1 -qps 0 -n 20 -loglevel Warning http://httpbin:8000/status/500 Running this will give you simulate 20 requests returning 500 . But if you have applied the circuit breaker if should allow just the one request as 500 and remaining requests should be tripped and a 503 should be returned . This doesn't happen.

jeson114 avatar May 29 '18 12:05 jeson114

Can someone at least validate this ?

jeson114 avatar Jun 07 '18 07:06 jeson114

@ldemailly Does fortio recreate connections transparently for HTTP ?

louiscryan avatar Jun 13 '18 21:06 louiscryan

@louiscryan fortio reports the exact number of sockets it had to open, if connections get closed in the middle, the number reported is not equal to the input -c connections (e.g "SocketCount": 48 in https://fortio.istio.io/data/qps_max-s2_to_s1-0.7.1-2018-04-05-22-04.json ) also any http error is reported in status (so 500 vs 503 are reported exactly)

ldemailly avatar Jun 13 '18 22:06 ldemailly

This is not fortio specific . I have encountered it with all east-west traffic while applying simplecb

jeson114 avatar Jun 14 '18 08:06 jeson114