kong
kong copied to clipboard
URI is sometimes `/kong_error_handler` on upstream status 502s
Is there an existing issue for this?
- [X] I have searched the existing issues
Kong version ($ kong version
)
kong:3.4.2-ubuntu
Current Behavior
In our logs some requests for which the upstream status is 502 have their $uri
variable overridden so that when Kong logs this request the log line appears to have /kong_error_handler
as the URI. This is seen in the log line below (I have removed some of the log fields).
2024-01-01 00:00:00.000
{
"uri": "/kong_error_handler",
"status": "502",
"upstream_status": "502",
"upstream_response_time": "0.001",
"upstream_response_length": "0",
}
Which corresponds to this log format
nginx_http_log_format: "main escape=json '{ \"uri\": \"$uri\", \"status\": \"$status\", \"upstream_status\": \"$upstream_status\", \"upstream_response_time\": \"$upstream_response_time\", \"upstream_response_length\": \"$upstream_response_length\"}'"
This is also reflected on the Prometheus metrics since these 502s which appear to originate from the upstream but have the $uri
parameter overridden appear on the kong_http_requests_total
metric as having the source as kong
rather than the service
.
It seems that the upstream does NOT return 502 for these requests (the request never reaches the upstream).
Expected Behavior
The URI should match the $uri
field should match the request URI (something like api/v1/things
). The $upstream_status
should not be 502.
Steps To Reproduce
This error happens intermittently and I cannot reproduce it.
Anything else?
We are trying to create alerts based on errors that originate from Kong. However, this bug prevents us from using the kong_http_requests_total
metric to create alerts.
Thanks for reporting @fv316. This indeed sounds like unintended behavior. We'll discuss this internally and come back to you.
@jschmid1 , do we have any update?