Pods with multiple containers not authenticating properly with default-image-pull-secret on 0.4.3

[hhollenstain-soda]

Setup:

• GCP
• Artifacts registry (gcp)
• set --default-image-pull-secret
• set --default-image-pull-secret-namespace
• running 0.4.3

This setup works on all but two deployments. The only notable difference between pods/rs working vs non working pods/rs if there is multiple defined containers. To note running 0.4.2/0.4.0 works. From looking at the changes in 0.4.3 it appears logic around registry secrets are handled has changed.

  Warning  FailedCreate      17m   replicaset-controller  Error creating: Internal error occurred: failed calling webhook "pods.kube-secrets-init.admission.doit-intl.com": an error on the server ("{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1beta1\",\"response\":{\"uid\":\"88fceb5d-b4a1-437b-b602-b578bf037b07\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"could not mutate object: failed to mutate pod: : cannot fetch image descriptor: GET https://us-docker.pkg.dev/v2/token?scope=repository%3A<project>%2F<repository>%2F<image>%3Apull\\u0026service=us-docker.pkg.dev: DENIED: Permission \\\"artifactregistry.repositories.downloadArtifacts\\\" denied on resource \\\"projects/<project>/locations/us/repositories/<repository>\\\" (or it may not exist)\"}}}") has prevented the request from succeeding

[hhollenstain]

Quick follow-up to note switching this over to utilize IAM/kuberenets IAM with GCP since the registry library switch from containerregistry to opencontainers and we bypassed this issue. Ultimately this is the ideal setup but wished the default-image-pull-secret didn't get impacted with this upgrade.