Istio Creating Replica Failure and blocking kubernetes deployment rollouts

[meSATYA]

Is this the right place to submit this?

• [X] This is not a security vulnerability or a crashing bug
• [X] This is not a question about how to use Istio

Bug Description

We faced Kubernetes "ReplicaFailure" issue as the istio-proxy sidecar was not running and the deployment status was showing as "ReplicaFailure". The issue starting without any new deployment to the istiod or istiobase and it came "out of nowhere" while there was change to the Kubernetes cluster happened. The pods didn't start as the isto-proxy sidecar was not starting and giving error "Error creating: admission webhook "namespace. sidecar-injector.istio.io" denied the request: failed to run injection template: failed to parse values.yaml; check Istiod logs for errors".

This happened for the second time and there is no workaround to fix this issue. Restarted the istiod pod but didn't solve the issue. While ran the below commands to check cacert for mutatingwebhook and istio-ca-root-cert and both the certificates matching content wise, though the md5sum doesn't match. Refer to the attached screenshot for certificate comparison.

At the end, after rolling out a blank deployment to the Istiod deployment via argocd fixed the issue. But, the root cause is not yet known. Need your assistance to get the root cause so that this issue can be avoided later.

Version

$ kubectl version                                                                                                                                                                                
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.4

istio-pilot: 1.23.2
istio-proxyv2: 1.23.2
Istio Helm Chart Version: 1.23.2

Additional Information

sh-3.2$ kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4

I1116 19:23:44.598878   73674 helpers.go:246] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server rejected our request for an unknown reason",
  "reason": "BadRequest",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "no body found"
      }
    ]
  },
  "code": 400
}]
Error from server (BadRequest): the server rejected our request for an unknown reason