Timeout errors due to some issue with the validating webhook

[Mais316]

Hi All,

We have hnc version 1.0.0 and our customers reported that they can no longer create any ns, checking the controller logs we can see

{"level":"error","ts":1688395630.3355894,"logger":"hierarchyconfig.reconcile","msg":"while updating apiserver","rid":3,"ns":"hnc-xyz","error":"Internal error occurred: failed calling webhook \"hierarchyconfigurations.hnc.x-k8s.io\": failed to call webhook: Post \"https://hnc-webhook-service.hnc-system.svc:443/validate-hnc-x-k8s-io-v1alpha2-hierarchyconfigurations?timeout=30s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
{"level":"error","ts":1688395630.335623,"logger":"controller.hierarchyconfiguration","msg":"Reconciler error","reconciler group":"hnc.x-k8s.io","reconciler kind":"HierarchyConfiguration","name":"hierarchy","namespace":"hnc-xyz","error":"Internal error occurred: failed calling webhook \"hierarchyconfigurations.hnc.x-k8s.io\": failed to call webhook: Post \"https://hnc-webhook-service.hnc-system.svc:443/validate-hnc-x-k8s-io-v1alpha2-hierarchyconfigurations?timeout=30s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
{"level":"error","ts":1688395630.3359044,"logger":"hierarchyconfig.reconcile","msg":"while updating apiserver","rid":1,"ns":"hnc-abc","error":"Internal error occurred: failed calling webhook \"objects.hnc.x-k8s.io\": failed to call webhook: Post \"https://hnc-webhook-service.hnc-system.svc:443/validate-objects?timeout=30s\": context deadline exceeded"}
{"level":"error","ts":1688395630.3359256,"logger":"controller.hierarchyconfiguration","msg":"Reconciler error","reconciler group":"hnc.x-k8s.io","reconciler kind":"HierarchyConfiguration","name":"hierarchy","namespace":"hnc-abc","error":"Internal error occurred: failed calling webhook \"objects.hnc.x-k8s.io\": failed to call webhook: Post \"https://hnc-webhook-service.hnc-system.svc:443/validate-objects?timeout=30s\": context deadline exceeded"}

We tried below, nothing helped

• Increasing the resources
• timeouts

We check calico logs all looks find, then tried port-forward and we can curl to the service. Kubernetes version is 1.124.12 Cluster is aws-eks, managed by Rancher 2.7.3

Appreciate your help as we are blocked, workaround is to remove the validation webhook, but that is not a solution.