kots icon indicating copy to clipboard operation
kots copied to clipboard

webhook that depends on a service is created first, and fails

Open allanyung opened this issue 5 years ago • 2 comments

We have a deployment that includes cert-manager.

When kots applies the configuration it does the webhooks before the services. Unfortunately, the cert-manager webhook depends on the service existing or it raises the following error

Error from server (InternalError): error when creating "/tmp/525695000": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: service "cert-manager-webhook" not found

This is unfortunate, because if you leave the cluster to it's own devices it will retry after the service is created and resolve itself. But the deploy is marked as failed in the customer kots dashboard.

This failure only happens on the first installation on a cluster. Subsequent deployments succeed, but in the absence of a retry option, you need to create a nonsense commit to trigger a new Release.

Ideally the webhook would get created after the service, but I appreciate this has big implications and may well cause other issues. Alternatively it would be nice to be able to swallow that error or have a mechanism to retry the deploy.

  • The stdout from the dashboard shows the ordering image

allanyung avatar May 21 '20 13:05 allanyung

Per @allanyung this might be fixed in an upcoming version of kustomize:

https://github.com/kubernetes-sigs/kustomize/pull/1104

https://github.com/kubernetes-sigs/kustomize/pull/2459

dexhorthy avatar May 21 '20 15:05 dexhorthy

kustomize 3.6.1 has been released. I've tested it locally and can confirm that the webhook is now ordered at the end of the generated output

allanyung avatar May 29 '20 07:05 allanyung