ReInvocation not mutating for new containers in a pod

[luciano-nbs]

What happened: We deployed a vault-agent-injector into our cluster which injects a vault-agent container into our pods via another mutating webhook.

When we deploy an application with the vault-agent annotations and a AWS annotation on the Service Account the main container has the AWS Environment vars injected as expected. Then the vault-agent-injector injects the vault-agent container into the pod as expected. However the AWS Environment vars aren't injected into this new container.

What you expected to happen: We expect the AWS webhook to run again after vault-agent-injector webhook and inject the AWS Environment vars.

How to reproduce it (as minimally and precisely as possible):

• Setup OIDC, SA with AWS role, etc (as normal)
• Setup vault-agent-injector (https://www.vaultproject.io/docs/platform/k8s/injector)
• Setup deployment with the following definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: vault-aws-test
  name: vault-aws-test
  namespace: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault-aws-test
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: vault-aws-test
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-foo-secret: secrets/my-app-secret
        vault.hashicorp.com/agent-inject-status: update
        vault.hashicorp.com/agent-pre-populate-only: "true"
    spec:
      serviceAccountName: my-app
      containers:
      - image: nginx
        name: nginx
        resources: {}
status: {}

Anything else we need to know?: When looking at the events for mutations on the pod we see the following:

        "mutation.webhook.admission.k8s.io/round_0_index_2": "{\"configuration\":\"pod-identity-webhook\",\"webhook\":\"iam-for-pods.amazonaws.com\",\"mutated\":true}",
        "mutation.webhook.admission.k8s.io/round_0_index_4": "{\"configuration\":\"vault-sidecar-injector-agent-injector-cfg\",\"webhook\":\"vault.hashicorp.com\",\"mutated\":true}",
        "mutation.webhook.admission.k8s.io/round_1_index_2": "{\"configuration\":\"pod-identity-webhook\",\"webhook\":\"iam-for-pods.amazonaws.com\",\"mutated\":false}",

However, we are unsure why the 2nd invocation of pod-identity-webhook didn't mutate the pod.

Environment:

• AWS Region: eu-west-2
• EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.2
• Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): 1.18
• Webhook Version: