Webhooks get context deadline exceeded

[mbrancato]

Pulling this in from Slack for tracking - After some time I've been unable to reliably resolve this. I am running the VM Operator on 6+ cluster but only getting this frequently on a few clusters, and I could not identify and correlation as to why.

After upgrading to v0.47.3 (helm chart 0.34.7) using the helm chart, I'm getting these errors from server-side apply:

Internal error occurred: failed calling webhook "vmagent.victoriametrics.com": failed to call webhook: Post "https://victoria-metrics-operator.monitoring.svc/validate-operator-victoriametrics-com-v1beta1-vmagent?timeout=10s": context deadline exceeded

I tried disabling the vmagent webhook, and then I started to get the same error with other resources like vmalert. There are no errors in the logs that seem directly related.

I've been able to confirm that we can connect to the service, it just seems to not respond to the webhook posts.