Webhook stops injecting Env Vars or Volumes/VolumeMounts on previously working deployment

[maximethebault]

What happened:

Pod-identity-webhook suddently stopped mutating pods for a deployment. In API server log, I can see the webhook is executed but results in mutating: false.

Recreating the pods was not enough to fix the issue. Recreating the deployment (without any other change) fixed the issue. In API server log, I can now see mutating: true.

What you expected to happen:

Pod-identity-webhook always works when no configuration changes was done.

How to reproduce it (as minimally and precisely as possible):

No idea. Maybe linked to a cluster upgrade?

Anything else we need to know?:

This is forked from a similar issue #174

This comment mentions the same issue: https://github.com/aws/amazon-eks-pod-identity-webhook/issues/174#issuecomment-1734325467

I wish I had more logs to go through, especially logs from the pod-identity-webhook, but they don't seem to be available. Any idea of how I could get them?

Environment:

• AWS Region: eu-west-1
• EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.6
• Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): 1.26
• Webhook Version: whatever comes with that combo