Getting 503 intermittently on ingress gateway

[ukumar155]

Title: Getting 503 intermittently on ingress gateway

Description:

We are using ISTIO as service mesh. We are seeing "response_code":503 intermittently on our ingress gateway.

Note: We have applied WASM filter- customfilter on ingress gateway.

When we enabled trace logs we can see below for all the flows which are giving 503:

2024-04-01T14:38:10.133142Z trace envoy http external/envoy/source/common/http/filter_manager.cc:539 [C458447][S4288012434896823960] decode headers called: filter=istio.metadata_exchange status=0 thread=63
2024-04-01T14:38:10.133153Z debug envoy http external/envoy/source/common/http/filter_manager.cc:917 [C458447][S4288012434896823960] Preparing local reply with details wasm_fail_stream thread=63
2024-04-01T14:38:10.133166Z debug envoy http external/envoy/source/common/http/filter_manager.cc:959 [C458447][S4288012434896823960] Executing sending local reply. thread=63
2024-04-01T14:38:10.133173Z trace envoy http external/envoy/source/common/http/filter_manager.cc:1129 [C458447][S4288012434896823960] encode headers called: filter=istio.stats status=0 thread=63
2024-04-01T14:38:10.133176Z trace envoy http external/envoy/source/common/http/filter_manager.cc:1129 [C458447][S4288012434896823960] encode headers called: filter=envoy.filters.http.cors status=0 thread=63
2024-04-01T14:38:10.133183Z trace envoy http external/envoy/source/common/http/filter_manager.cc:1129 [C458447][S4288012434896823960] encode headers called: filter=envoy.filters.http.fault status=0 thread=63
2024-04-01T14:38:10.133184Z trace envoy http external/envoy/source/common/http/filter_manager.cc:1129 [C458447][S4288012434896823960] encode headers called: filter=istio-system.customfilter status=4 thread=63
2024-04-01T14:38:10.133187Z trace envoy http external/envoy/source/common/http/filter_manager.cc:530 [C458447][S4288012434896823960] decodeHeaders filter iteration aborted due to local reply: filter=istio-system.customfilter thread=63
2024-04-01T14:38:10.133191Z trace envoy http external/envoy/source/common/http/filter_manager.cc:539 [C458447][S4288012434896823960] decode headers called: filter=istio-system.customfilter status=1 thread=63
2024-04-01T14:38:10.133194Z trace envoy http2 external/envoy/source/common/http/http2/codec_impl.cc:1002 [C458447] dispatched 1446 bytes thread=63


Envoy version - v1.25

cc: @zhaohuabing 

[RyanTheOptimist]

"wasm_fail_stream" seems to be called when the wasm VM fails, fwiw.

CC: @mpwarres @lizan

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.