aws-application-networking-k8s icon indicating copy to clipboard operation
aws-application-networking-k8s copied to clipboard

Controller is not following `GatewayClass` spec, finalizer must be added

Open aaroniscode opened this issue 2 years ago • 1 comments

According to the GatewayClass spec:

Whenever one or more Gateways are using a GatewayClass, implementations MUST add the gateway-exists-finalizer.gateway.networking.k8s.io finalizer on the associated GatewayClass. This ensures that a GatewayClass associated with a Gateway is not deleted while in use.

I noticed the finalizer was not present and I'm able to delete the GatewayClass while it's in use.

» kubectl get gateway my-hotel
NAME       CLASS                ADDRESS   READY   AGE
my-hotel   amazon-vpc-lattice                     161m
» kubectl delete gatewayclass amazon-vpc-lattice
gatewayclass.gateway.networking.k8s.io "amazon-vpc-lattice" deleted

But now the system is in an unstable state. I'm unable to delete HTTPRoute resources.

aaroniscode avatar Feb 25 '23 02:02 aaroniscode

So did some digging here and it appears this is a bit controversial upstream and this spec requirement may be removed, see https://github.com/kubernetes-sigs/gateway-api/issues/1411

I couldn't find any other project that has implemented a finalizer for GatewayClass. In fact, a number of projects have decided not to implement:

  • Istio declined to support, https://github.com/istio/istio/issues/34128
  • Contour added them and removed them due to issues, https://github.com/projectcontour/contour/issues/4732
  • Kong is waiting for resolution of upstream issue mentioned above, https://github.com/Kong/kubernetes-ingress-controller/issues/2984

Finalizers are generally disliked because they block deletion and should be reserved for instances where they make the most sense, like for our HTTPRoute and Gateway resources because they represent VPC Lattice Resources that need to be cleaned up when deleted.

Plus, implementing this would likely mean that deleting the GatewayClass would mean deleting all ServiceNetworks and all the dependent Services which would be highly destructive and unlikely the intent of someone deleting the GatewayClass.

Recommend we put this on hold and wait for resolution of upstream. Best guess it appears upstream is likely to remove the finalizer requirement from the spec.

aaroniscode avatar Mar 05 '23 15:03 aaroniscode