vault injection breaking after k8s 1.22 with preemption

[rwong2888]

I have vault injector with 3 replicas on preemptible nodes. I've upgraded to k8s 1.22 on one cluster and am noticing an issue with the injector.

The application deployment is not getting the vault sidecar injected and crash looping. Killing the app pod does not inject vault. Killing the 3 running vault pods and then killing the app pod does not inject vault. It isn't until I kill all the pods in the vault namespace (NodeAffinity,Completed,Error,Running) that the app pods can be reinjected.

K8s Version: v1.22.8-gke.202 Helm Chart: v0.20.1 Injector Image Tag: 0.16.1 Injector Agent Image Tag: 1.10.3

sidecar-injector Using internal leader elector logic for webhook certificate management
sidecar-injector E0712 14:01:30.181268       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://xxx.xxx.xxx.xxx:443/api/v1/namespaces/vault/secrets?l
sidecar-injector Listening on ":8080"...
sidecar-injector 2022-07-12T14:01:31.500Z [INFO]  handler: Starting handler..
sidecar-injector 2022-07-12T14:01:31.600Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
sidecar-injector 2022-07-12T14:01:31.601Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
sidecar-injector 2022-07-12T14:01:31.601Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
sidecar-injector 2022-07-12T14:01:31.601Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
sidecar-injector 2022-07-12T14:01:31.601Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
sidecar-injector 2022-07-12T14:01:31.601Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
sidecar-injector 2022-07-12T14:01:31.601Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
sidecar-injector I0712 14:01:32.531554       1 request.go:665] Waited for 1.045250766s due to client-side throttling, not priority and fairness, request: GET:https://xxx.xxx.xxx.xxx:443/apis/networking.istio.io/v1alpha3?timeout=32s
sidecar-injector 2022-07-12T14:02:55.368Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
sidecar-injector 2022-07-12T14:02:56.760Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
sidecar-injector 2022-07-13T02:23:49.962Z [ERROR] handler: http: TLS handshake error from 127.0.0.6:56923: remote error: tls: bad certificate
sidecar-injector 2022-07-13T02:23:50.025Z [ERROR] handler: http: TLS handshake error from 127.0.0.6:49969: remote error: tls: bad certificate
sidecar-injector 2022-07-13T02:23:53.685Z [ERROR] handler: http: TLS handshake error from 127.0.0.6:37439: remote error: tls: bad certificate

[bzupnick]

Experiencing the same thing for ~ a week.

K8s Version: v1.21.12-eks-a64ea69 Helm Chart: v0.20.1 Injector Image Tag: 0.16.1 Injector Agent Image Tag: 1.10.3

☝️ 55 and 145 restarts

[rwong2888]

Thanks @bzupnick , I'm not actually seeing the injector crash loop, but rather the application pods reliant upon vault injector are not getting the vault sidecar injected, thus leading to the application itself crash looping. So it may be a different issue from what you are experiencing.

[rwong2888]

As a workaround, I've put vault-injector on a non-preemptible nodepool