Merge gateway EnvoyProxy parameters between the class and gateway instance

[plnordquist]

Description: When using a set of EnvoyProxy parameters at the gateway class level with spec.parametersRef and a separate set of parameters at the gateway level with spec.infrastructure.parametersRef, it would be nice if the configuration was merged and the gateway parameters override the gateway class parameters. For example, I want to setup the gateway class to set a default replica count of two and a PDB with minAvailable=1 for all my gateways and setup my gateways with specific loadBalancerIPs.

The Gateway API does call this feature out as implementation specific and includes a recommendation for this support but through testing of Envoy Gateway v1.1.0 gateway instance parameters seem to override all gateway class parameters.

Relevant Links: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.GatewayClassSpec https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.GatewayInfrastructure

[arkodg]

wdyt @envoyproxy/gateway-maintainers @envoyproxy/gateway-reviewers

[zirain]

Replace is good to me, an option to support Merge in EnvoyGateway is reasonable to me.

[cnvergence]

this is a good idea

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[jalaziz]

Just wanted to offer some more examples of why this would be useful.

Given that the Gateway API is designed to satisfy different personas and most of the customization happens at the EnvoyProxy level, being able to merge would enable different personas to control different aspects of the final Envoy configuration.

In our case, we would want to set tracing, metrics, logging, and PDB configuration at the GatewayClass level, which our infrastructure team provisions as a shared resource. However, our application teams also need to set service annotations for the AWS load balancer controller. Some of these annotations can be shared, but others are very much app specific.

Overall, I think the merge w/ overwrite behavior would make the most sense. However, certain fields like annotations and labels might require special thought and care. For example, should an empty set of service annotations at the Gateway level erase annotations at the GatewayClass level?

[arkodg]

thanks @jalaziz this makes sense, as a workaround, you could set EnvoyProxy at the GatewayClass level and use Gateway.Spec.Infrastaructure.Annotations to solve your use case https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.GatewayInfrastructure

[jalaziz]

thanks @jalaziz this makes sense, as a workaround, you could set EnvoyProxy at the GatewayClass level and use Gateway.Spec.Infrastaructure.Annotations to solve your use case https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.GatewayInfrastructure

Yup, that's something I considered but was trying to avoid cause then the service account and other resources end up with load balancer annotations. That should be fine, just not ideal.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

BTP introduced a MergeType field to allow Route level BTP to merge into GW level BTP https://github.com/envoyproxy/gateway/blob/ec94c9848d88d430624396f70ac6a0e8124d3420/api/v1alpha1/backendtrafficpolicy_types.go#L50 something similar can be done here

[wimnat]

I think the most complicated factor here is deciding on precedence and control of overwriting config.

As an example:

@jalaziz states above:

However, our application teams also need to set service annotations for the AWS load balancer controller.

But I actual favour the opposite - I don't want an application team to be able to override the AWS load balancer controller annotations and provision themselves an internet-facing LB.

Perhaps an override policy for specific fields could be defined within the EnvoyProxy config and then GatewayClass EnvoyProxy override policy would have precedence over the same config defined in EnvoyProxy referred to from a Gateway.

[arkodg]

imo more complicated policy authorization can be enforced using tools like Kyverno https://kyverno.io/docs/policy-types/cluster-policy/validate/ and stretches the scope of this project

[plnordquist]

Personally I would rather have the Gateway parameters overwrite configuration from the GatewayClass without any rules. From the perspective of the 3 tier Gateway API Security Model, Application Developers creating and updating gateways is not something that should be allowed, see https://gateway-api.sigs.k8s.io/concepts/security-model/. It looks like there is also a 4 tier security model there where Application Admins may be given privileges to create or update a gateway and in that case there are many tools to control the level of access they have like Kyverno or Validating Admission Policy or Gatekeeper, etc.