Gateway `ExternalIP` assignement

[fad3t]

Good morning,

I have a GatewayClass with an EnvoyProxy config that sets mergeGateways to true and assigns an ExternalIP to the corresponding service.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
# ...
spec:
  mergeGateways: true
  provider:
    kubernetes:
      envoyService:
        externalTrafficPolicy: Local
        loadBalancerClass: io.cilium/bgp-control-plane
        patch:
          type: StrategicMerge
          value:
            spec:
              externalIPs:
              - 40.40.40.1
        type: LoadBalancer
    type: Kubernetes

When a user creates a Gateway, we expect external-dns to use the ExternalIP of the gateway as target. However, it's using the address mentioned in the gateway Status, which refers to the LB IP:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
# ...
status:
  addresses:
  - type: IPAddress
    value: 10.10.10.3

Some time ago I was told I can work around by setting the Spec.Addresses field on the Gateway. It works fine, although it actually doesn't care about the configured value, as it would read it from the Service if it exists.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
# ...
spec:
  addresses:
  - type: IPAddress
    value: 1.2.3.4
# ...
status:
  addresses:
  - type: IPAddress
    value: 40.40.40.1

I'm wondering if this could be reworked to make it a bit more intuitive -- happy to contribute if need be. In my case, I'm managing the GatewayClass and users are responsible for deploying their own Gateway -- so I don't really want to tell them they have to put a dummy IP (this is confusing) or the public IP (which varies depending on the env/site/..) in their manifests.

I'm thinking about something like this:

1. if the gateway has an address defined, use this one (ofc some validation is required here)
2. then, use the external IP of the service if it exists
3. otherwise, reuse the existing logic depending on the service type (LB, ClusterIP, ..)

FYI the logic comes from this piece of code: https://github.com/envoyproxy/gateway/blob/main/internal/gatewayapi/status/gateway.go#L55

Any feedback or idea is welcome, thanks!

[buroa]

Not sure if this helps, but this is how we do it: https://github.com/buroa/k8s-gitops/blob/main/kubernetes/apps/networking/envoy-gateway/config/envoy.yaml#L57

[fad3t]

Thanks @buroa - we actually don't have issues with the LB IP (we also use this annotation btw). It is really about the external IP that is used by external-dns to map the DNS record to the public IP of the Envoy gateway.

[jukie]

Isn't this expected default behavior for external-dns? It looks to be configurable via annotations https://github.com/kubernetes-sigs/external-dns/blob/master/docs/annotations/annotations.md#external-dnsalphakubernetesioaccess

[fad3t]

Isn't this expected default behavior for external-dns? It looks to be configurable via annotations https://github.com/kubernetes-sigs/external-dns/blob/master/docs/annotations/annotations.md#external-dnsalphakubernetesioaccess

I didn't configure this annotation, so it should be set to public by default. Anyway I don't think this annotation is read from Gateway API resources (docs say only the target annotation is read). External DNS reads the addresses from the status of the gateway (cf. https://github.com/kubernetes-sigs/external-dns/blob/master/source/gateway.go#L383) unless there's an override (= setting the target annotation, which I don't want to do) -- that's why I'm trying to have the external IP of the gateway reflected in its status.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[fad3t]

Any feedback on the proposed logic?

1. if the gateway has an address defined, use this one (ofc some validation is required here)
2. then, use the external IP of the service if it exists
3. otherwise, reuse the existing logic depending on the service type (LB, ClusterIP, ..)

[arkodg]

is there any good justification/prior art to prioritize external ip over load balancer ip in status ?

[fad3t]

• external tooling such as External DNS use the status field to set the DNS record value, so if a Gateway has both a private and a public IP address, I think it'd make more sense to use the public one -- but that's just my opinion;
• the current logic already prioritizes the External IP, but only when the Address field is set. I find this counter-intuitive, as the value of the Address field is ignored anyway.

[gregsidelinger]

I stumbled across this issue while trying to set a custom Hostname in the gateway address spec. Which it turns out is not currently supported since if addresses is set in the gateway spec you always pull the IPs from the service, even if I specify "type: Hostname" in the address spec. It would be nice if the controller actually honored the settings in the gateway spec instead of assuming I wanted an IP.

I agree with @fad3t as the current implementation when addresses is set does not make any sense. In fact I would say the controller does the opposite of a user wants since it always ignores the addresses value and puts the service's IP on the status.