Kong Gateway API with 104 Connection reset by peer message to OpenShift IngressController

[jrosal06]

In OpenShift cluster we have installed Kong Gateway API, after the cluster upgrede from OpenShift 4.12 (Kubernetes 1.25) to OpenShift 4.14 (Kubernetes 1.27) that was carried out to this cluster it is observed that it is sending the following error:

2024/07/18 18:14:31 [info] 1634#0: *14053057 client 10.127.4.2 closed keepalive connection (104: Connection reset by peer)

2024/07/21 18:10:48 [info] 2606#0: *38884119 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too (104: Connection reset by peer) while connecting to upstream, client: 10.127.4.2, server: kong, request: "GET /provisioning-services/api/health HTTP/1.1", upstream: "http://10.123.135.29:1084/provisioning-services/api/health", host: "xxxxxxxxx"

And that in it the ingress controller is updated from Haproxy 2.4 to Haproxy 2.6.

The confusing part here is that, in the cluster with version 4.14, it doesn't show FIN, ACK TCP packet before closing the connection (RST), but it does in the cluster with version 4.12. We are talking about the application having the same version (Kong 2.5) in both clusters but we only see this atypical behavior in the cluster with version 4.14. There must be some change you aren't identifying at the HAProxy level. It is fundamental to find the justification for why it behaves this way in different versions of OpenShift.

`Before:

"No.","Time","Source","Destination","Protocol","Length","Info" "516","11:38:21,386258","10.126.2.2","10.125.7.54","TCP","80","58248 > 8080 [SYN] Seq=4129447575 Win=26580 Len=0 MSS=8860 SACK_PERM TSval=3126154924 TSecr=0 WS=128" "517","11:38:21,386510","10.125.7.54","10.126.2.2","TCP","80","8080 > 58248 [SYN, ACK] Seq=3249228591 Ack=4129447576 Win=26544 Len=0 MSS=8860 SACK_PERM TSval=1266477712 TSecr=3126154924 WS=128" "518","11:38:21,386768","10.126.2.2","10.125.7.54","TCP","72","58248 > 8080 [ACK] Seq=4129447576 Ack=3249228592 Win=26624 Len=0 TSval=3126154924 TSecr=1266477712" "519","11:38:21,386813","10.126.2.2","10.125.7.54","HTTP","387","GET /provisioning-services/api/health HTTP/1.1 " "520","11:38:21,386849","10.125.7.54","10.126.2.2","TCP","72","8080 > 58248 [ACK] Seq=3249228592 Ack=4129447891 Win=27648 Len=0 TSval=1266477712 TSecr=3126154924" "736","11:38:22,455572","10.125.7.54","10.126.2.2","HTTP/JSON","637","HTTP/1.1 200 OK , JSON (application/json)" "737","11:38:22,455873","10.126.2.2","10.125.7.54","TCP","72","58248 > 8080 [ACK] Seq=4129447891 Ack=3249229157 Win=27776 Len=0 TSval=3126155993 TSecr=1266478781" "738","11:38:22,456098","10.126.2.2","10.125.7.54","TCP","72","58248 > 8080 [FIN, ACK] Seq=4129447891 Ack=3249229157 Win=27776 Len=0 TSval=3126155994 TSecr=1266478781" "739","11:38:22,456144","10.125.7.54","10.126.2.2","TCP","72","8080 > 58248 [FIN, ACK] Seq=3249229157 Ack=4129447892 Win=27648 Len=0 TSval=1266478781 TSecr=3126155994" "740","11:38:22,456266","10.126.2.2","10.125.7.54","TCP","72","58248 > 8080 [RST, ACK] Seq=4129447892 Ack=3249229157 Win=27776 Len=0 TSval=3126155994 TSecr=1266478781" "741","11:38:22,456569","10.126.2.2","10.125.7.54","TCP","60","58248 > 8080 [RST] Seq=4129447892 Win=0 Len=0"

After

"No.","Time","Source","Destination","Protocol","Length","Info" "607","11:40:43,122819","10.126.2.2","10.124.7.57","TCP","80","47892 > 8080 [SYN] Seq=2905734366 Win=62020 Len=0 MSS=8860 SACK_PERM TSval=4260484801 TSecr=0 WS=128" "608","11:40:43,122852","10.124.7.57","10.126.2.2","TCP","80","8080 > 47892 [SYN, ACK] Seq=2819640922 Ack=2905734367 Win=61936 Len=0 MSS=8860 SACK_PERM TSval=679465162 TSecr=4260484801 WS=128" "609","11:40:43,123024","10.126.2.2","10.124.7.57","TCP","72","47892 > 8080 [ACK] Seq=2905734367 Ack=2819640923 Win=62080 Len=0 TSval=4260484801 TSecr=679465162" "610","11:40:43,123128","10.126.2.2","10.124.7.57","HTTP","387","GET /provisioning-services/api/health HTTP/1.1 " "611","11:40:43,123136","10.124.7.57","10.126.2.2","TCP","72","8080 > 47892 [ACK] Seq=2819640923 Ack=2905734682 Win=61696 Len=0 TSval=679465162 TSecr=4260484801" "857","11:40:44,173392","10.124.7.57","10.126.2.2","HTTP/JSON","637","HTTP/1.1 200 OK , JSON (application/json)" "858","11:40:44,173589","10.126.2.2","10.124.7.57","TCP","72","47892 > 8080 [ACK] Seq=2905734682 Ack=2819641488 Win=61568 Len=0 TSval=4260485852 TSecr=679466213" "966","11:40:48,092830","10.126.2.2","10.124.7.57","TCP","72","47892 > 8080 [RST, ACK] Seq=2905734682 Ack=2819641488 Win=61568 Len=0 TSval=4260489771 TSecr=679466213"`