hoot icon indicating copy to clipboard operation
hoot copied to clipboard

episode suggestion: Istio Rate Limit is working even if ratelimit & redis pods are down - Azure AKS 1.21.9

Open zohebs341 opened this issue 3 years ago • 1 comments

Kindly refer attachment. I've deployed rate limit along with Redis in Azure AKS 1.21.9 and did some basic tests related to rate limiting, it worked as expected.

Issue Description:

I scaled down rate limit & Redis pods and tested rate limit functionality. As the rate limit and Redis pods are down, rate limit functionality should not work. But in my case, the rate limit worked even if ratelimit/Redis pods are down.

The expectation is it should not work right? If it is working, how come?

Azure AKS 1.21.9

Istio Version - 1.14

Documents used for ratelimit deployment:

https://istio.io/latest/docs/tasks/policy-enforcement/rate-limit/#verify-local-rate-limit https://github.com/istio/istio/blob/release-1.14/samples/ratelimit/rate-limit-service.yaml poc-ratelimitdown.docx

zohebs341 avatar Aug 13 '22 06:08 zohebs341

it shouldn't work. depending on the filter config, it may have failed open. To make it fail close, change failure_mode_deny to true. https://github.com/envoyproxy/envoy/blob/0c282b9c6deb93dbc26c5a98ba0056dfebfe05f9/api/envoy/extensions/filters/http/ratelimit/v3/rate_limit.proto#L71

yuval-k avatar Feb 27 '23 18:02 yuval-k