cilium Ingress: No healthy upstream for headless service

Is there an existing issue for this?

[X] I have searched the existing issues

Reported by: @knfoo

What happened?

If the Ingress is configured with a headless service like below, the response from the Ingress route is coming back as no healthy upstream. Ideally, the request should be coming IP (which is specified in Endpoint object of the headless Service)

apiVersion: v1
kind: Service
metadata:
  name: details-headless
  namespace: default
spec:
  ports:
  - name: http
    port: 9080
    protocol: TCP
    targetPort: 9080
  clusterIP: None
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
---
apiVersion: v1
kind: Endpoints
metadata:
  name: details-headless
  namespace: default
subsets:
- addresses:
  - ip: 10.244.0.95 # manually configure this Endpoints resource with valid backend IP
  ports:
  - name: http
    port: 9080
    protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: basic-ingress-headless
  namespace: default
spec:
  ingressClassName: cilium
  rules:
  - http:
      paths:
      - backend:
          service:
            name: details-headless
            port:
              number: 9080
        path: /details
        pathType: Prefix
---

Cilium Version

1.13.x 1.12.x

Kernel Version

N/A

Kubernetes Version

N/A

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Jan 19 '23 11:01 sayboras

Upon investigating this issue, I found out that endpoint IP is not bumped to Envoy due to below code block, which skipped further processing of Service.

https://github.com/cilium/cilium/blob/07dea5d1da0337ff2e5c27ef993a1e8f5fa41047/pkg/k8s/watchers/watcher.go#L902-L904

Seems like it's a side effect of having shared code between Service watcher and Envoy

https://github.com/cilium/cilium/issues/19698

Move Envoy specifics from pkg/service to pkg/k8s/watchers, see https://github.com/cilium/cilium/pull/18894#discussion_r840633249

The workaround is to add dummy Service IP like below.

apiVersion: v1
kind: Service
metadata:
  name: details-headless
  namespace: default
spec:
  ports:
  - name: http
    port: 9080
    protocol: TCP
    targetPort: 9080
  clusterIP: 192.192.192.192 --> dummy IP, this should belong to service CIDR of cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack

Jan 19 '23 11:01 sayboras

Should be fixed by #25105 ?

May 01 '23 12:05 youngnick

yes, I am marking the above PR as draft to add e2e test.

May 02 '23 05:05 sayboras

@sayboras I'm hitting this problem on the latest v1.15.5. Are there any plans to get this addressed?

cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.852][14][debug][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:137] tls:onServerName(), requestedServerName: my.example.org
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.852][14][debug][misc] [cilium/bpf_metadata.cc:297] EGRESS POD IP: 192.168.111.36, destination IP: 192.168.111.238
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.852][14][debug][filter] [cilium/conntrack.cc:178] cilium.bpf_metadata: Using conntrack map global
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.852][14][debug][filter] [cilium/conntrack.cc:229] cilium.bpf_metadata: IPv4 conntrack map global lookup failed: No such file or directory
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.852][14][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 192.168.111.36 has ID 2
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.852][14][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 192.168.111.238 has ID 2
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.853][14][debug][filter] [cilium/ipcache.cc:90] cilium.ipcache: 10.244.0.27 has ID 8
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.853][14][debug][filter] [./cilium/socket_option.h:239] Cilium SocketOption(): source_identity: 8, ingress: false, port: 443, pod_ip: 10.244.0.27, source_addresses: /10.244.0.27:0/, mark: 80b00 (magic mark: b00, cluster: 0, ID: 8), proxy_id: 0
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.853][14][debug][filter] [cilium/network_filter.cc:76] cilium.network: onNewConnection
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.853][14][debug][conn_handler] [external/envoy/source/extensions/listener_managers/listener_manager/active_tcp_listener.cc:159] [Tags: "ConnectionId":"8"] new connection from 192.168.111.36:51930
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.862][14][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:391] [Tags: "ConnectionId":"8"] new stream
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.862][14][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1194] [Tags: "ConnectionId":"8","StreamId":"4800741104043154596"] request headers complete (end_stream=true):
cilium-envoy-5c5bq cilium-envoy ':authority', 'my.example.org'
cilium-envoy-5c5bq cilium-envoy ':path', '/mmgtest/'
cilium-envoy-5c5bq cilium-envoy ':method', 'GET'
cilium-envoy-5c5bq cilium-envoy 'user-agent', 'curl/8.7.1'
cilium-envoy-5c5bq cilium-envoy 'accept', '*/*'
cilium-envoy-5c5bq cilium-envoy 
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.862][14][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1177] [Tags: "ConnectionId":"8","StreamId":"4800741104043154596"] request end stream
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.862][14][debug][connection] [external/envoy/source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"8"] current connecting state: false
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.862][14][debug][router] [external/envoy/source/common/router/router.cc:520] [Tags: "ConnectionId":"8","StreamId":"4800741104043154596"] cluster 'public-gateway/cilium-gateway-public/mattermost:mmtatu:8065' match for URL '/mmgtest/'
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.862][14][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1942] no healthy host for HTTP connection pool
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.863][14][debug][http] [external/envoy/source/common/http/filter_manager.cc:985] [Tags: "ConnectionId":"8","StreamId":"4800741104043154596"] Preparing local reply with details no_healthy_upstream
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.863][14][debug][http] [external/envoy/source/common/http/filter_manager.cc:1027] [Tags: "ConnectionId":"8","StreamId":"4800741104043154596"] Executing sending local reply.
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.863][14][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1863] [Tags: "ConnectionId":"8","StreamId":"4800741104043154596"] encoding headers via codec (end_stream=false):
cilium-envoy-5c5bq cilium-envoy ':status', '503'
cilium-envoy-5c5bq cilium-envoy 'content-length', '19'
cilium-envoy-5c5bq cilium-envoy 'content-type', 'text/plain'
cilium-envoy-5c5bq cilium-envoy 'date', 'Mon, 20 May 2024 16:15:03 GMT'
cilium-envoy-5c5bq cilium-envoy 'server', 'envoy'
cilium-envoy-5c5bq cilium-envoy 
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.863][14][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1968] [Tags: "ConnectionId":"8","StreamId":"4800741104043154596"] Codec completed encoding stream.
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.864][14][debug][connection] [external/envoy/source/common/network/connection_impl.cc:714] [Tags: "ConnectionId":"8"] remote close
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.864][14][debug][connection] [external/envoy/source/common/network/connection_impl.cc:278] [Tags: "ConnectionId":"8"] closing socket: 0
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.864][14][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:368] [Tags: "ConnectionId":"8"] SSL shutdown: rc=1
cilium-envoy-5c5bq cilium-envoy [2024-05-20 16:15:03.866][14][debug][conn_handler] [external/envoy/source/extensions/listener_managers/listener_manager/active_stream_listener_base.cc:135] [Tags: "ConnectionId":"8"] adding to cleanup list

Service

$ kubectl get svc mmtatu -o yaml | kubectl neat                                                    
apiVersion: v1
kind: Service
metadata:
  labels:
    app: mattermost
    installation.mattermost.com/installation: mmtatu
    installation.mattermost.com/resource: mmtatu
  name: mmtatu
  namespace: mattermost
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: app
    port: 8065
    targetPort: app
  - appProtocol: http
    name: metrics
    port: 8067
    targetPort: metrics
  selector:
    app: mattermost
    installation.mattermost.com/installation: mmtatu
    installation.mattermost.com/resource: mmtatu

httproute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mmtatu-mattermost-public
  namespace: mattermost
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: public
    namespace: public-gateway
    sectionName: https-main
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: mmtatu
      port: 8065
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /mmgtest/

Whenever creating a new service with clusterIP and modifying the route it worked.

May 20 '24 16:05 tchellomello

Thanks for your comment, the approach in mentioned PR is no longer ideal due to recent refactor and discussion.

I am currently exploring alternative options now.

May 21 '24 00:05 sayboras

@sayboras I appreciate the update. I'll be tuned in and please let me know how I can help testing.

May 21 '24 03:05 tchellomello

Thanks, can you help to test out the changes in https://github.com/cilium/cilium/pull/32644, the image digests can be found in the below CI job:

https://github.com/cilium/cilium/actions/runs/9203017338/job/25314240955?pr=32644

May 23 '24 06:05 sayboras

@sayboras just to make sure I'll test it properly.

I need to replace the container image "cilium" by the "cilium-ci", right?

Do I need to replace the operator, clustermesh and all other images too?

Furthermore, I'm running envoy as a dedicated daemonset, but I don't see in that CI job the cilium-envoy. Is that OK?

May 25 '24 14:05 tchellomello

I need to replace the container image "cilium" by the "cilium-ci", right? Do I need to replace the operator, clustermesh and all other images too?

Yes, you are right. For this change, the operator is not required to be updated, however, no harm to keep them in sync.

Furthermore, I'm running envoy as a dedicated daemonset, but I don't see in that CI job the cilium-envoy. Is that OK?

Cilium Envoy is built from cilium/proxy repo, no change is required.

May 26 '24 11:05 sayboras

I'll test it tonight and provide an update @sayboras

May 28 '24 17:05 tchellomello

@sayboras I was going to post on the PR but decided to do here instead. Not sure exactly the reason but this is pretty odd. I'm using a BGP Announcements on my cluster and as soon as changed the operator and the cilium images to CI, I cannot connect to the my HTTP Gateway (however other services exposed via Loadbalancer and shared via BGP are working). It seems something is up with the envoy gateway directly. Check it out:

Images and status

cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       OK
    \__/       ClusterMesh:        OK

DaemonSet              cilium-envoy             Desired: 4, Ready: 4/4, Available: 4/4
Deployment             hubble-ui                Desired: 1, Ready: 1/1, Available: 1/1
Deployment             cilium-operator          Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-relay             Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium                   Desired: 4, Ready: 4/4, Available: 4/4
Deployment             clustermesh-apiserver    Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium-envoy             Running: 4
                       hubble-ui                Running: 1
                       clustermesh-apiserver    Running: 1
                       cilium-operator          Running: 1
                       hubble-relay             Running: 1
                       cilium                   Running: 4
Cluster Pods:          139/139 managed by Cilium
Helm chart version:    
Image versions         hubble-ui                quay.io/cilium/hubble-ui:v0.13.0@sha256:7d663dc16538dd6e29061abd1047013a645e6e69c115e008bee9ea9fef9a6666: 1
                       hubble-ui                quay.io/cilium/hubble-ui-backend:v0.13.0@sha256:1e7657d997c5a48253bb8dc91ecee75b63018d16ff5e5797e5af367336bc8803: 1
                       clustermesh-apiserver    quay.io/cilium/clustermesh-apiserver:v1.15.5@sha256:914549caf4376a844b5e7696019182dd2a655b89d6a3cad10f9d0f9821759fd7: 2
                       cilium-operator          quay.io/cilium/operator-generic-ci:a83a327fd6663778cb3af62fb9254f05ef66206d@sha256:5edbb65701aec94e81c89613c0275b9584d7c73897f7c3185f4b6ce44e85fd53: 1
                       hubble-relay             quay.io/cilium/hubble-relay:v1.15.5@sha256:1d24b24e3477ccf9b5ad081827db635419c136a2bd84a3e60f37b26a38dd0781: 1
                       cilium                   quay.io/cilium/cilium-ci:a83a327fd6663778cb3af62fb9254f05ef66206d@sha256:8a6169c94371ec59ac4f3c431dc7c2606a37678b2398498d907a9f18928f0c7d: 4
                       cilium-envoy             quay.io/cilium/cilium-envoy:v1.28.3-31ec52ec5f2e4d28a8e19a0bfb872fa48cf7a515@sha256:bc8dcc3bc008e3a5aab98edb73a0985e6ef9469bda49d5bb3004c001c995c380: 4

Gateway up

kubectl get gateway -n public-gateway 
NAME     CLASS    ADDRESS         PROGRAMMED   AGE
public   cilium   192.168.108.7   True         10d

kubectl get svc -n public-gateway 
NAME                    TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                      AGE
cilium-gateway-public   LoadBalancer   10.105.54.2   192.168.108.7   443:32005/TCP,80:30343/TCP   10d

BPG routes correct

cilium  bgp routes 
(Defaulting to `available ipv4 unicast` routes, please see help for more options)

Node               VRouter   Prefix              NextHop   Age     Attrs
p53.tatu.home      64552     192.168.108.1/32    0.0.0.0   8m13s   [{Origin: i} {Nexthop: 0.0.0.0}]   
  
                   64552     192.168.108.4/32    0.0.0.0   8m13s   [{Origin: i} {Nexthop: 0.0.0.0}]   

                   64552     192.168.108.7/32    0.0.0.0   8m13s   [{Origin: i} {Nexthop: 0.0.0.0}]   
 
                   64552     192.168.108.9/32    0.0.0.0   8m13s   [{Origin: i} {Nexthop: 0.0.0.0}]   
p70.tatu.home      64552     192.168.108.1/32    0.0.0.0   7m52s   [{Origin: i} {Nexthop: 0.0.0.0}]     
 
                   64552     192.168.108.7/32    0.0.0.0   7m52s   [{Origin: i} {Nexthop: 0.0.0.0}]   
  
t470n1.tatu.home   64552     192.168.108.1/32    0.0.0.0   7m48s   [{Origin: i} {Nexthop: 0.0.0.0}]   
                   64552     192.168.108.7/32    0.0.0.0   7m48s   [{Origin: i} {Nexthop: 0.0.0.0}]   
  
t470n2.tatu.home   64552     192.168.108.1/32    0.0.0.0   8m11s   [{Origin: i} {Nexthop: 0.0.0.0}]    
                   64552     192.168.108.7/32    0.0.0.0   8m11s   [{Origin: i} {Nexthop: 0.0.0.0}]

Other services exposed working

❯ kubectl get svc -n default postgresql-toca
NAME              TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
postgresql-toca   LoadBalancer   10.108.198.10   192.168.108.4   5432:30557/TCP   5d21h

❯ psql -h 192.168.108.4
psql (16.2, server 15.2 (Debian 15.2-1.pgdg110+1))
Type "help" for help.

postgres=#

HTTPRoute is accepted

kubectl describe httproutes.gateway.networking.k8s.io  -n mattermost mattermost-public
Name:         mattermost-public
Namespace:    mattermost
Labels:       app.kubernetes.io/instance=mattermost
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1
Kind:         HTTPRoute
Metadata:
  Creation Timestamp:  2024-05-20T16:33:47Z
  Generation:          7
  Resource Version:    857653081
  UID:                 9164c8f7-3b7b-4877-ba85-7515e5df1ab4
Spec:
  Parent Refs:
    Group:         gateway.networking.k8s.io
    Kind:          Gateway
    Name:          public
    Namespace:     public-gateway
    Section Name:  https-main
  Rules:
    Backend Refs:
      Group:   
      Kind:    Service
      Name:    mmtatu-workaround
      Port:    8065
      Weight:  1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /mmgtest
Status:
  Parents:
    Conditions:
      Last Transition Time:  2024-05-25T14:43:30Z
      Message:               Accepted HTTPRoute
      Observed Generation:   7
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-05-25T14:43:30Z
      Message:               Service reference is valid
      Observed Generation:   7
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         io.cilium/gateway-controller
    Parent Ref:
      Group:         gateway.networking.k8s.io
      Kind:          Gateway
      Name:          public
      Namespace:     public-gateway
      Section Name:  https-main
Events:              <none>

But connection is failing to reach out TCP 443

nc -vv 192.168.108.7 443
nc: connect to 192.168.108.7 port 443 (tcp) failed: Connection refused

curl -v https://internal.host/mmgtest/
* Host internal.host:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.108.7
*   Trying 192.168.108.7:443...
* connect to 192.168.108.7 port 443 from 192.168.111.58 port 52498 failed: Connection refused
* Failed to connect to internal.host port 443 after 20 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to internal.host port 443 after 20 ms: Couldn't connect to server

I also disabled the envoy as a dedicated daemonset and the result was the same.

However, restoring back the original image works again (however, the problem with the headless service remains as expected).

Any ideas?

May 30 '24 04:05 tchellomello

:wave:

Most likely it's related to BGP Announcements IMO, the CI tests are actually having the similar setup, can you cross check if there is any discrepancy which could cause issue :thinking:

https://github.com/cilium/cilium/blob/c52bee974efe8ae9196355c5ab44d7278468c6d3/.github/workflows/conformance-ingress.yaml#L223

Jun 07 '24 11:06 sayboras

@sayboras :wave:

Yes, I can confirm the issue was related with L2 announcements. I had to disable STP on my switches to get the ARP announcements to work. I was hitting several TCP Spurious Retransmissions but it seems to be better now. Thanks for working on that fix! :1st_place_medal:

Jun 12 '24 13:06 tchellomello

cilium cilium copied to clipboard

Ingress: No healthy upstream for headless service

Is there an existing issue for this?

What happened?

Cilium Version

Kernel Version

Kubernetes Version

Sysdump

Relevant log output

Anything else?

Code of Conduct

cilium
cilium copied to clipboard