hoot
hoot copied to clipboard
episode suggestion: Istio Rate Limit is working even if ratelimit & redis pods are down - Azure AKS 1.21.9
Kindly refer attachment. I've deployed rate limit along with Redis in Azure AKS 1.21.9 and did some basic tests related to rate limiting, it worked as expected.
Issue Description:
I scaled down rate limit & Redis pods and tested rate limit functionality. As the rate limit and Redis pods are down, rate limit functionality should not work. But in my case, the rate limit worked even if ratelimit/Redis pods are down.
The expectation is it should not work right? If it is working, how come?
Azure AKS 1.21.9
Istio Version - 1.14
Documents used for ratelimit deployment:
https://istio.io/latest/docs/tasks/policy-enforcement/rate-limit/#verify-local-rate-limit https://github.com/istio/istio/blob/release-1.14/samples/ratelimit/rate-limit-service.yaml poc-ratelimitdown.docx
it shouldn't work. depending on the filter config, it may have failed open. To make it fail close, change failure_mode_deny to true.
https://github.com/envoyproxy/envoy/blob/0c282b9c6deb93dbc26c5a98ba0056dfebfe05f9/api/envoy/extensions/filters/http/ratelimit/v3/rate_limit.proto#L71