gateway

gateway copied to clipboard

Gateway API conformance defines a base set of 3 Gateways, all using the same GatewayClass: https://github.com/kubernetes-sigs/gateway-api/blob/main/conformance/base/manifests.yaml#L16-L62

EG's design is to be 1:1 with a GatewayClass, with a single address for all Gateways for that class. As such, all Listeners across all of the Gateways need to be compatible/collapsed, per https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1beta1/gateway_types.go#L72-L106.

Unfortunately, the Gateways defined for conformance are not collapsible: they each specify an HTTP Listener on port 80 with no specific hostname. EG is going to see those as conflicting, and not be able to program them. However, these Gateways are definitely valid, according to Gateway API. There's no requirement that Listeners across Gateways of a given GatewayClass be collapsible.

Seems like there are a few ways we could deal with this:

• consider these Gateways' Listeners compatible/collapsible -- this seems problematic, would mean all the routes across all the Gateways would get munged together
• give each Gateway a unique address
• something else?

cc @youngnick, would appreciate your thoughts on this since I know you have historical context re: merging Listeners across Gateways.

@youngnick can you please review to make sure ^ is accurate?

@skriss thanks again for bringing this issue to light. IMHO this is a bug in the spec, so I created https://github.com/kubernetes-sigs/gateway-api/issues/1385. Let's see what the community has to say.

The way I read it, the spec is fine -- it gives implementors the option to merge compatible Listeners across Gateways, but does not imply that they must, nor that Listeners across Gateways must be compatible. In this case they're not compatible, but they're still perfectly valid specs, so we need to handle them differently.

It seems to me like the best path forward is to go with a separate LB service per Gateway, each of which forwards to a different set of Envoy ports. That way, each Gateway has its own address/exposed ports, and that address only exposes the routes attached to that Gateway, but they all still share the same underlying data plane as originally designed.

We could also explore whether it's possible to selectively merge Gateways into a single LB service, when their Listeners are compatible. The challenge I see with this approach is what was alluded to on https://github.com/kubernetes-sigs/gateway-api/issues/1385 -- namely, as Listeners are added/modified on a Gateway over time, they may become incompatible with another Gateway's Listeners that they were previously compatible with, which would require us to then separate them out into separate LBs, which seems problematic for users (addresses are changing, how does cutover happen? etc).

Yeah, this point is kind of making me rethink the "GatewayClass = Envoy set" idea, and that it may be better to have each Gateway be a separate LB service and Envoy deployment. Particularly if that's a common pattern in other implementations, being too different is only going to create more problems for us in the future.

I definitely agree that having the address a service is served out of change because someone added a Listener somewhere that made two Gateways not compatible is very bad UX.

From the call:

What Arko suggested on today's call is along the lines of what Steve posted; that we have one Envoy and have a Service per Gateway that forwards to a different set of Envoy ports. That we sort the Listeners, and then assign them ports in order, in order to be deterministic.

Just sorting the Listeners is problematic; if A and C exist, then you create B, then C's port assignment would change. We don't necessarily need to be deterministic, we need to be stable.

I suggested storing the assigned port number in to the status. Arko points out that this would make the translator non-stateless.

---

As I think about it more, there are hacks we could do like taking a 16-bit hash of the listener and using that as the port number... but we'd still have to deal with possibility of conflicts, and so we'd have to fall back to a persistent state. Writing the assignments back to Kube is the only robust solution if we want use a single Envoy deployment. It's either that, or an Envoy per Gateway.

We wouldn't have to have in-process state for the translator; the state could 100% live in Kubernetes. But it would mean that we need functional writing of statuses; which we should have soon for Kubernetes, but might not even make sense for other config sources.

thanks for capturing the discussion @LukeShu

• the other option of creating a unique Envoy Deployment per Gateway would require changes in
  • Gateway API translator - it would need to create an xDS IR per Gateway
  • EG would need to map a Gateway to a Envoy ID
  • xDS Server would need to be aware of this mapping

@arkodg in addition to the points you make above, is it correct to say that an SDS sidecar is required per Envoy to properly manage secrets across the different data planes?

@arkodg in addition to the points you make above, is it correct to say that an SDS sidecar is required per Envoy to properly manage secrets across the different data planes?

I think simple TLS for auth and Service Account Token for authz could be a solution. I dont think this changes the control plane authn authz design rather it changes how the xds-server decides which node-id (which could be the gateway name) should serve which snapshot https://github.com/envoyproxy/gateway/blob/ac5ea770c7c7b7b227832e89cec9d7efaf1c5b36/internal/xds/cache/snapshotcache.go#L275

We discussed this on the call today, the outcome is this:

• there is a lot of support for shifting our model to run a separate xds server container deployment per Gateway, coordinated by the infrastruture manager, essentially splitting the infrastructure manager and xds server into separate commands/binaries.
• @arkodg is going to spend a couple of days seeing if he can get a more shared configuration working though.
• If we do need to move to separate infra-manager and control plane, then we will pivot v0.2.0's goal to be about delivering a good experience for the Gateway API features we have now, updating docs etc, to document what features and limitations we currently have, and then put a redesign into scope for v0.3.0.