Support routing to Service Cluster IP

[arkodg]

[arkodg]

two ways to solve this

• add a field within EnvoyProxy called disableEndpointRouting / enableClusterIPRouting which allows us to fallback to routing using Service Cluster IP
• incorporate this into the load balancing API

[arkodg]

ptal @envoyproxy/gateway-maintainers

[arkodg]

chatted with @kflynn the other day about this and we both feel adding it in the load balancing API is more suitable

• if we added it another API, userX may enable serviceIP at the EnvoyProxy level, user Y may set the LB Policy and then wonder why the LB policy isn't working. Adding it inside the LB Policy forces the user to be aware of the lb & routing decision to endpoints

[Xunzhuo]

I am +1 on load balancing api, if we set this in EnvoyProxy, IMHO, this is on GC level and working for all managed Gateways, but if people just want this to work in specific Gateways, a Policy is more suitable.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

this new ClusterIP load balancing type should live in https://github.com/envoyproxy/gateway/blob/main/api/v1alpha1/loadbalancer_types.go

[deszhou]

Hi @arkodg, i think this use case can be achieved by adding a fallbackToClusterIP field

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  namespace: default
  name: policy-for-route1
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: httproute-1
    namespace: default
  loadBalancer:
    type: LeastRequest
    slowStart:
      window: 300s
    fallbackToClusterIP: true # Enables fallback to service's cluster IP

[arkodg]

@yeedove, I personally prefer

  loadBalancer:
    type: ClusterIP

[kflynn]

Turns out that to get Envoy Gateway working well with Linkerd, we either need this or we need an l5d-dst-override header to be added to the request, so I would definitely like this. 🙂

I'd personally vote for routingType: service or routingType: endpoint, but I'm good with @arkodg's type suggestion.

[deszhou]

please assign to me

[arkodg]

I like @kflynn's suggestion of a dedicated field routingType, if we take this approach, we'll have to use CEL to disable loadBalancing if routingType: Service

[arkodg]

• we discussed this in the community meeting today, most were in favor of this field living in the BackendTrafficPolicy thanks for volunteering to pick this up @kflynn, temporarily assigning this to you for now here's a link to help you get started https://github.com/envoyproxy/gateway/blob/2a38de6067d421026994b8cbea202e399e68c98e/internal/gatewayapi/route.go#L1074 (you'll need to override the global EndpointRoutingDisabled field)

[arkodg]

rethinking this one, if the field lives in BackendTrafficPolicy, then it won't apply to other backendRefs in other places, e.g. ext_auth in SecurityPolicy, or ext_proc in EnvoyExtensionPolicy or openTelemetry sinks in EnvoyProxy . we may want to move this field into EnvoyProxy cc @zhaohuabing @guydc

[zhaohuabing]

I guess this depends on do we need per-service configuration for the choice of cluster IP vs pod IP?

[arkodg]

one reason to select Service routing is for mesh case (sidecar beside gateway) to work seamlessly, so it will work for the xRoute backends (with sidecars ) but not if the ext auth / ext proc / Otel (have sidecars ) because it won't get intercepted properly in the gateway sidecar

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[jasonmoore2k]

Any progress on this? I’m looking to deploy EG alongside an Istio mesh and from what I’m hearing here and in the related ticket, I would need this feature to make it work?

[arkodg]

@kflynn still planning on working on this one ?

[ryanhristovski]

@kflynn by setting the routingType: service would we lose the observability into each endpoint in the /clusters config & metrics and only display the ClusterIP?

@arkodg discussed the possibility of running an idle linkerd proxy sidecar on the gateway which would handle mTLS, but then it seems like we'd lose Linkerd multi-cluster service discovery.

[lnattrass]

@kflynn by setting the routingType: service would we lose the observability into each endpoint in the /clusters config & metrics and only display the ClusterIP?

I think if service-mesh is being used, the metrics are likely going to be more useful from the mesh than from the ingress, because the mesh will handle service -> service as well as envoy -> service in the same format and namespace..

[kflynn]

I think @lnattrass has the right of it – part of the point of the mesh is that it should be handling both metrics and routing for you in this case.

[kflynn]

@arkodg I'd still like to, yes! so here's a question for you. There already is an EndpointRoutingDisabled field in the Translator; do we think that works as a global switch? or is a leftover untested vestigial thing? 🙂

[arkodg]

@arkodg I'd still like to, yes! so here's a question for you. There already is an EndpointRoutingDisabled field in the Translator; do we think that works as a global switch? or is a leftover untested vestigial thing? 🙂

@kflynn this should work as expected, the gatewayapi test should prove this easily, if the flag is set, the clusterIP will get set it in the IR

[ryanhristovski]

@lnattrass we're not necessarily interested in Linkerd for routing on our ingress, but more so for cross-cluster service discovery and mTLS. So we'd prefer to keep the load balancing capabilities and metrics as close to our proxy as possible. In the past the CPU requirements of running 4k RPS+ on an ingress pod with linkerd over doubles our usage and since we've always used headless services we haven't relied on Linkerd's load balancing.

I suppose this is more of a niche use-case, but it does limit us on implementing multi-cluster Linkerd.

@kflynn would Linkerd be open to mirroring EndpointSlices as well or is it out of scope? I could open an issue (and potentially work on it) if you think it's worth considering

[evacchi]

Hi @kflynn I'm sorry to take this over from you, but we internally needed a fix earlier :) feel free to check out and comment my PR, though!

[kflynn]

@evacchi No apology needed! as you can see, I've been pulled onto other things, thank you for making this happen!! 🙂