Intermittently returns 503 when routing to ExternalName service (same-cluster, cross-namespace)

[jeromeguiard]

Is there an existing issue for this?

• [x] I have searched the existing issues

Kong version ($ kong version)

Kong 3.8

Current Behavior

In Kong 3.8 deployed on EKS, when using a Kubernetes ExternalName service (resolving to another service in a different namespace within the same cluster), Kong intermittently returns 503 Service Unavailable responses—about 50% of the time—when routing traffic via a Kubernetes Ingress or a Gateway API HTTPRoute.

If the HTTPRoute uses a BackendRef that directly targets the actual service in the other namespace, bypassing the ExternalName indirection, the issue does not occur.

Also curl the target service hostname manually from the Kong pod do not show a 503 at all.

While setting kong in debug I do not observe any specific message:

2025/05/22 16:23:31 [debug] 1406#0: *12680 [lua] init.lua:1397: balancer(): setting address (try 1): 172.20.159.184:80
2025/05/22 16:23:31 [debug] 1406#0: *12680 [lua] init.lua:1430: balancer(): enabled connection keepalive (pool=172.20.159.184|80, pool_size=512, idle_timeout=60, max_requests=10000)
10.250.11.54 - - [22/May/2025:16:23:31 +0000] "POST /api HTTP/1.1" 503 107 "-" "curl/8.7.1" kong_request_id: "f7b91609241e6716bb55a1636c5ba46f"

This is the log request I can see from the tries it identifies the correct ip of the service in the targeted namespace

{
  "client_ip": "121.200.241.162",
  "started_at": 1747931011686,
  "route": {
    "service": {
      "id": "8942cb25-df0d-5569-bbb6-2f64b5d12ea2"
    },
    "path_handling": "v0",
    "paths": [
      "~/api($|/)(.*)"
    ],
    "id": "270b826d-38cd-5536-a7cd-a317480ced93",
    "preserve_host": true,
    "strip_path": false,
    "regex_priority": 0,
    "name": "xxx.xxx-xxxxx-xxx-ingress.ooo-vendor.sei-www.myhost.com.80",
    "hosts": [
      "www.myhost.com"
    ],
    "request_buffering": true,
    "response_buffering": true,
    "tags": [
      "k8s-name:myingress-ingress",
      "k8s-namespace:mnamespace",
      "k8s-kind:Ingress",
      "k8s-uid:6430d3b3-f5a5-40f7-8e81-879f271a8af4",
      "k8s-group:networking.k8s.io",
      "k8s-version:v1"
    ],
    "https_redirect_status_code": 426,
    "ws_id": "0dc6f45b-8f8d-40d2-a504-473544ee190b",
    "created_at": 1747930762,
    "updated_at": 1747930762,
    "protocols": [
      "http",
      "https"
    ]
  },
  "response": {
    "size": 400,
    "headers": {
      "server": "kong/3.8.0",
      "via": "1.1 kong/3.8.0",
      "content-length": "107",
      "x-kong-upstream-latency": "0",
      "x-kong-request-id": "f7b91609241e6716bb55a1636c5ba46f",
      "content-type": "text/html; charset=UTF-8",
      "connection": "close",
      "cache-control": "no-cache",
      "x-kong-proxy-latency": "0"
    },
    "status": 503
  },
  "kong_plugin": "my-file-log",
  "workspace_name": "default",
  "upstream_uri": "/",
  "tries": [
    {
      "port": 80,
      "balancer_start": 1747931011686,
      "balancer_start_ns": 1747931011686800000,
      "balancer_latency_ns": 14592,
      "ip": "172.20.159.184",
      "balancer_latency": 0
    }
  ],
  "workspace": "0dc6f45b-8f8d-40d2-a504-473544ee190b",
  "source": "upstream",
  "service": {
    "write_timeout": 60000,
    "enabled": true,
    "tags": [
      "k8s-name:service-vendor",
      "k8s-namespace:mnamespace",
      "k8s-kind:Service",
      "k8s-uid:720a4962-0339-4d64-b782-486d63825550",
      "k8s-version:v1"
    ],
    "id": "8942cb25-df0d-5569-bbb6-2f64b5d12ea2",
    "port": 80,
    "name": "mnamespace.service-vendor.80",
    "connect_timeout": 60000,
    "read_timeout": 60000,
    "host": "service-vendor.80.svc",
    "created_at": 1747930762,
    "updated_at": 1747930762,
    "retries": 5,
    "path": "/",
    "protocol": "http",
    "ws_id": "0dc6f45b-8f8d-40d2-a504-473544ee190b"
  },
  "upstream_status": "503",
  "latencies": {
    "kong": 0,
    "proxy": 0,
    "receive": 1,
    "request": 1
  },
  "request": {
    "uri": "/api",
    "querystring": {},
    "tls": {
      "cipher": "ECDHE-ECDSA-AES128-GCM-SHA256",
      "client_verify": "NONE",
      "version": "TLSv1.2"
    },
    "url": "http://www.myhost.com:80/api",
    "size": 392,
    "headers": {
      "user-agent": "curl/8.7.1",
      "accept": "*/*",
      "content-length": "91",
      "host": "www.myhost.com",
      "content-type": "application/json"
    },
    "method": "POST",
    "id": "f7b91609241e6716bb55a1636c5ba46f"
  }
}

Expected Behavior

Kong should route all requests to the target service without intermittent failure.

Steps To Reproduce

service.yaml config

apiVersion: v1
kind: Service
metadata:
  name: service-vendor
spec:
  externalName: m_service.m_namespace.svc.cluster.local.
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  sessionAffinity: None
  type: ExternalName

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    konghq.com/plugins: transformer-path
    konghq.com/regex-prefix: /~
  name: myingress
spec:
  ingressClassName: kong
  rules:
    - host: www.myhost.com
      http:
        paths:
          - backend:
              service:
                name: service-vendor
                port:
                  number: 8545
            path: /~/api($|/)(.*)
            pathType: ImplementationSpecific

Anything else?

No response

[isc30]

Hi, facing the same issue here