azure-key-vault-to-kubernetes [BUG] envinjector unable to pull docker image from azurecr using aadpodidbinding

Versions Controller version: spvest/azure-keyvault-controller:1.3.2 Env-Injector version: spvest/azure-keyvault-env:1.3.1 Helm chart version: akv2k8s-2.2.2 Helm App Version: 1.3.1 AKS Version: 1.23.12

Description When attempting to use env-injection with custom user assigned managed identity using aadpodidbinding the env-injector fails to pull down the docker image with the below error message

2022/10/05 20:02:50 [ERROR] admission webhook error: failed to get auto cmd, error: GET https://dwtestregistry.azurecr.io/oauth2/token?scope=repository%3Asecret-printer%3Apull&service=dwtestregistry.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.
cannot fetch image descriptor

The MI has been given AcrPull access and assigned to the cluster with:

az aks pod-identity add \
  --resource-group dw-akv2k8s-test \
  --cluster-name dw-akv2k8s-test-aks-01 \
  --namespace akv2k8s \
  --name akv2k8s \
  --identity-resource-id /subscriptions/<<subscription>>/resourcegroups/dw-akv2k8s-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dw-test-akv2k8s-mi

and the akv2k8s helm values are:

env:
  AZURE_ENVIRONMENT: "AzureCloud"
controller:
  keyVaultAuth: environment
  podLabels:
    aadpodidbinding: akv2k8s
env_injector:
  keyVaultAuth: environment
  podLabels:
    aadpodidbinding: akv2k8s

To Reproduce

Create a standard AKS cluster with managed identity enabled
Create MI for akv2k8s with: az identity create -g dw-akv2k8s-test -n dw-test-akv2k8s-mi
Assign AcrPull to eh acr: az role assignment create --assignee <<>> --role <<>> --scope <<>>
Grant secret GET to the AKV: az keyvault set-policy --name <<>> --object-id <<>> --secret-permissions get list
Assign the MI to aks: az aks pod-identity add -g <<>> -cluster-name dw-akv2k8s-test-aks-01 --namespace akv2k8s --name akv2k8s --identity-resource-id <<>>
Deploy test

---
apiVersion: v1
kind: Namespace
metadata:
  name: "secret-printer"
  labels:
    azure-key-vault-env-injection: "enabled"
---
apiVersion: spv.no/v1alpha1
kind: AzureKeyVaultSecret
metadata:
  name: 'akv-mitest'
  namespace: 'secret-printer'
spec:
  vault:
    name: dw-akv2k8s-test-akv # name of key vault
    object:
      name: mitest    # name of the akv object
      type: secret    # akv object type
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: 'secret-printer'
  namespace: 'secret-printer'
spec:
  selector:
    matchLabels:
      app: 'secret-printer'
  template:
    metadata:
      labels:
        app: 'secret-printer'
    spec:
      containers:
        - name: secret-printer
          image: 'dwtestregistry.azurecr.io/secret-printer:latest'
          imagePullPolicy: Always
          env:
            - name: akv-mitest
              value: 'akv-mitest@azurekeyvault'

Expected behaviour

pod to start up
env_injector init container to be added
main container to have the secret injected into env

Logs

$ kube logs pod/akv2k8s-envinjector-79db7f56c4-8cw56 -n akv2k8s
I1005 19:05:17.657239       1 version.go:31] "version info" version="1.3.1" commit="11d9861" buildDate="2022-04-18T15:59:36Z" component="webhook"
I1005 19:05:17.657336       1 main.go:290] "active settings" httpPort="8080" httpPortExternal="80" tlsPort="8443" tlsPortExternal="443" mtlsPort="9443" mtlsPortExternal="9443" serveMetrics=false authType="environment" useAuthService=true dockerInspectionTimeout=20 cloudConfigPath="/etc/kubernetes/azure.json" logLevel="6" authServiceName="akv2k8s-envinjector" mtlsPortExternal="9443" mtlsPort="9443"
I1005 19:05:17.657736       1 main.go:350] "not using cloudConfig for auth - looking for azure key vault credentials in environment"
I1005 19:05:17.657782       1 provider.go:274] "azure: using managed identity extension to retrieve access token" id=""
I1005 19:05:17.657794       1 provider.go:289] "azure: using System Assigned MSI to retrieve access token" url="http://169.254.169.254/metadata/identity/oauth2/token"
I1005 19:05:17.657813       1 main.go:377] "checking credentials by getting authorizer"
I1005 19:05:18.684725       1 plugins.go:43] Registered credential provider "akv2k8s"
I1005 19:05:18.684771       1 auth.go:54] "auth service ca cert" file="/var/ca-cert/tls.crt"
I1005 19:05:18.684782       1 auth.go:55] "auth service ca key" file="/var/ca-cert/tls.key"
I1005 19:05:18.684940       1 main.go:393] "serving auth validation endpoint" path=":8080/auth/{namespace}/{pod}"
I1005 19:05:18.684967       1 main.go:397] "serving health endpoint" path=":8080/healthz"
I1005 19:05:18.685049       1 main.go:444] "serving encrypted auth endpoint" path=":9443/auth"
2022/10/05 19:05:18 [WARN] no tracer active
I1005 19:05:18.685254       1 main.go:420] "serving encrypted webhook endpoint" path=":8443/pods"
I1005 19:05:18.685277       1 main.go:423] "serving encrypted healthz endpoint" path=":8443/healthz"
2022/10/05 19:06:10 [DEBUG] reviewing request b37d0af6-dfe5-462e-a530-706d4eb8db83, named: secret-printer/
I1005 19:06:10.657034       1 main.go:142] "found pod to mutate" pod="secret-printer/"
I1005 19:06:10.657066       1 pod.go:285] "creating client certificate to use with auth service" ="(MISSING)"
I1005 19:06:10.657076       1 clientCert.go:25] "creating x509 key pair for ca cert and key"
I1005 19:06:10.657169       1 clientCert.go:32] "parse certificate"
I1005 19:06:10.657191       1 clientCert.go:38] "generating client key"
I1005 19:06:10.700828       1 clientCert.go:44] "generating serial number"
I1005 19:06:10.700866       1 clientCert.go:66] "crating x509 certificate"
I1005 19:06:10.702845       1 pod.go:292] "mutate init-containers" ="(MISSING)"
I1005 19:06:10.702861       1 pod.go:298] "mutate containers" ="(MISSING)"
I1005 19:06:10.702870       1 pod.go:116] "found container to mutate" container="secret-printer/secret-printer"
I1005 19:06:10.702879       1 pod.go:119] "checking for env vars to inject" container="secret-printer/secret-printer"
I1005 19:06:10.702891       1 pod.go:122] "found env var to inject" env="akv-mitest@azurekeyvault" container="secret-printer/secret-printer"
I1005 19:06:10.702900       1 registry.go:30] "getting container command for container" container="secret-printer/secret-printer"
I1005 19:06:10.702910       1 registry.go:36] "no cmd override in kubernetes for container, checking docker image configuration for entrypoint and cmd" image="dwtestregistry.azurecr.io/secret-printer:latest" container="secret-printer/secret-printer"
I1005 19:06:10.741759       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/secret-printer/serviceaccounts/default 200 OK in 38 milliseconds
I1005 19:06:10.741965       1 plugins.go:64] Registering credential provider: .dockercfg
I1005 19:06:10.741984       1 plugins.go:64] Registering credential provider: akv2k8s
I1005 19:06:10.741992       1 plugins.go:64] Registering credential provider: amazon-ecr
I1005 19:06:10.742011       1 azure_credentials.go:182] Azure config unspecified, disabling
I1005 19:06:10.742255       1 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
I1005 19:06:10.742308       1 config.go:144] looking for config.json at /config.json
I1005 19:06:10.742345       1 config.go:144] looking for config.json at /config.json
I1005 19:06:10.742357       1 config.go:144] looking for config.json at /root/.docker/config.json
I1005 19:06:10.742391       1 config.go:144] looking for config.json at /.docker/config.json
I1005 19:06:10.742477       1 config.go:110] looking for .dockercfg at /.dockercfg
I1005 19:06:10.742503       1 config.go:110] looking for .dockercfg at /.dockercfg
I1005 19:06:10.742514       1 config.go:110] looking for .dockercfg at /root/.dockercfg
I1005 19:06:10.742525       1 config.go:110] looking for .dockercfg at /.dockercfg
I1005 19:06:10.742541       1 provider.go:82] Docker config file not found: couldn't find valid .dockercfg after checking in [  /root /]
I1005 19:06:10.742565       1 provider.go:274] "azure: using managed identity extension to retrieve access token" id=""
I1005 19:06:10.742582       1 provider.go:289] "azure: using System Assigned MSI to retrieve access token" url="http://169.254.169.254/metadata/identity/oauth2/token"
I1005 19:06:10.751896       1 acr.go:154] "discovering auth redirects" url="dwtestregistry.azurecr.io"
I1005 19:06:10.779106       1 acr.go:160] exchanging an acr refresh_token
E1005 19:06:10.839539       1 acr.go:53] "failed to get acr credentials" err="failed to perform token exchange: Www-Authenticate: auth url https://dwtestregistry.azurecr.io/oauth2/exchange responded with status code 403"
E1005 19:06:10.854515       1 main.go:165] "failed to mutate" err=<
        failed to get auto cmd, error: GET https://dwtestregistry.azurecr.io/oauth2/token?scope=repository%3Asecret-printer%3Apull&service=dwtestregistry.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.
        cannot fetch image descriptor
        github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144
        github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(*Registry).GetImageConfig
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103
        main.getContainerCmd
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39
        main.podWebHook.mutateContainers
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143
        main.podWebHook.mutatePodSpec
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299
        main.vaultSecretsMutator
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163
        github.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate
                /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25
        github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview
                /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:128
        github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review
                /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:120
        github.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review
                /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/internal/instrumenting/instrumenting.go:42
        github.com/slok/kubewebhook/pkg/http.HandlerFor.func1
                /go/pkg/mod/github.com/slok/[email protected]/pkg/http/handler.go:64
        net/http.HandlerFunc.ServeHTTP
                /usr/local/go/src/net/http/server.go:2084
        github.com/gorilla/mux.(*Router).ServeHTTP
                /go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210
        net/http.serverHandler.ServeHTTP
                /usr/local/go/src/net/http/server.go:2916
        net/http.(*conn).serve
                /usr/local/go/src/net/http/server.go:1966
        runtime.goexit
                /usr/local/go/src/runtime/asm_amd64.s:1571
 > pod="secret-printer/"
2022/10/05 19:06:10 [ERROR] admission webhook error: failed to get auto cmd, error: GET https://dwtestregistry.azurecr.io/oauth2/token?scope=repository%3Asecret-printer%3Apull&service=dwtestregistry.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.
cannot fetch image descriptor
github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144
github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(*Registry).GetImageConfig
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103
main.getContainerCmd
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39
main.podWebHook.mutateContainers
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143
main.podWebHook.mutatePodSpec
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299
main.vaultSecretsMutator
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163
github.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate
        /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25
github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview
        /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:128
github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review
        /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:120
github.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review
        /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/internal/instrumenting/instrumenting.go:42
github.com/slok/kubewebhook/pkg/http.HandlerFor.func1
        /go/pkg/mod/github.com/slok/[email protected]/pkg/http/handler.go:64
net/http.HandlerFunc.ServeHTTP
        /usr/local/go/src/net/http/server.go:2084
github.com/gorilla/mux.(*Router).ServeHTTP
        /go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210
net/http.serverHandler.ServeHTTP
        /usr/local/go/src/net/http/server.go:2916
net/http.(*conn).serve
        /usr/local/go/src/net/http/server.go:1966
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1571

Replicaset event log

  Warning  FailedCreate  13m   replicaset-controller  Error creating: Internal error occurred: failed calling webhook "pods.env-injector.admission.spv.no": failed to call webhook: an error on the server ("{\"response\":{\"uid\":\"69e5bcef-efc4-4c23-97dd-97ee5c9d04d5\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"failed to get auto cmd, error: GET https://dwtestregistry.azurecr.io/oauth2/token?scope=repository%3Asecret-printer%3Apull\\u0026service=dwtestregistry.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\\ncannot fetch image descriptor\\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144\\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(*Registry).GetImageConfig\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103\\nmain.getContainerCmd\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39\\nmain.podWebHook.mutateContainers\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143\\nmain.podWebHook.mutatePodSpec\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299\\nmain.vaultSecretsMutator\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:128\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:120\\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/internal/") has prevented the request from succeeding

Additional context The end goal here is to get the env_injector to run as non-root and not hostPath, this means we can't simply use the azure.json

We are seeing the the AzureIdentity created successfully

$ kube get azureidentity -o yaml
apiVersion: v1
items:
- apiVersion: aadpodidentity.k8s.io/v1
  kind: AzureIdentity
  metadata:
    annotations:
     <<>>
    creationTimestamp: "2022-10-05T16:38:47Z"
    generation: 1
    labels:
      addonmanager.kubernetes.io/mode: Reconcile
      kubernetes.azure.com/managedby: aks
    name: akv2k8s
    namespace: akv2k8s
    resourceVersion: "129591"
    uid: 78bc8563-ba22-4b26-b52a-89bec2f63a51
  spec:
    clientID: <<correct id>>
    resourceID: <<correct resource id>>
    type: 0
kind: List
metadata:
  resourceVersion: ""

$ kube get azureidentitybinding -o yaml
apiVersion: v1
items:
- apiVersion: aadpodidentity.k8s.io/v1
  kind: AzureIdentityBinding
  metadata:
    annotations:
       <<>>
    creationTimestamp: "2022-10-05T16:38:47Z"
    generation: 1
    labels:
      addonmanager.kubernetes.io/mode: Reconcile
      kubernetes.azure.com/managedby: aks
    name: akv2k8s-binding
    namespace: akv2k8s
    resourceVersion: "129592"
    uid: e8e6e5e5-d440-4e24-8023-8bfc7ebd91e6
  spec:
    azureIdentity: akv2k8s
    selector: akv2k8s
kind: List
metadata:
  resourceVersion: ""

Oct 05 '22 21:10 dougalII

I assign the AcrPull role to my kubelet identity and do not have this problem.

Can you please try this also, maybe it is enough to just assign it the pod identity of the env_injector.

Feb 09 '23 12:02 cgroschupp

I'm havving a similar issue. Versions Env-Injector version: spvest/azure-keyvault-webhook:1.4.0 Helm chart version: akv2k8s-2.3.2 Helm App Version: 1.4.0 AKS Version: 1.23.12

Description In my aks cluster, without using akv2k8s the images from my private acr are pulled correctly. When I use akv2k8s, the image don't pull. The errors bellow:

I0308 14:47:56.996345       1 version.go:31] "version info" version="1.4.0" commit="15d87b2" buildDate="2022-12-08T21:19:10Z" component="webhook"
I0308 14:47:56.996432       1 main.go:290] "active settings" httpPort="8080" httpPortExternal="80" tlsPort="8443" tlsPortExternal="443" mtlsPort="9443" mtlsPortExternal="9443" serveMetrics=false authType="azureCloudConfig" useAuthService=true dockerInspectionTimeout=20 cloudConfigPath="/etc/kubernetes/azure.json" logLevel="4" authServiceName="akv2k8s-envinjector" mtlsPortExternal="9443" mtlsPort="9443"
I0308 14:47:56.997194       1 main.go:359] "using cloudConfig for auth - reading credentials" file="/etc/kubernetes/azure.json"
I0308 14:47:56.997902       1 provider.go:274] "azure: using managed identity extension to retrieve access token" id="xxxx0303-d490-xxxx-xxxx-xxxxxxxxxxxx"
I0308 14:47:56.997921       1 provider.go:281] "azure: using managed identity extension to retrieve access token" id="xxxx0303-d490-xxxx-xxxx-xxxxxxxxxxxx"
I0308 14:47:56.998000       1 main.go:377] "checking credentials by getting authorizer"
I0308 14:47:57.015834       1 plugins.go:43] Registered credential provider "akv2k8s"
I0308 14:47:57.015888       1 auth.go:54] "auth service ca cert" file="/var/ca-cert/tls.crt"
I0308 14:47:57.015912       1 auth.go:55] "auth service ca key" file="/var/ca-cert/tls.key"
I0308 14:47:57.016016       1 main.go:393] "serving auth validation endpoint" path=":8080/auth/{namespace}/{pod}"
I0308 14:47:57.016039       1 main.go:397] "serving health endpoint" path=":8080/healthz"
I0308 14:47:57.016109       1 main.go:444] "serving encrypted auth endpoint" path=":9443/auth"
2023/03/08 14:47:57 [WARN] no tracer active
I0308 14:47:57.016206       1 main.go:420] "serving encrypted webhook endpoint" path=":8443/pods"
I0308 14:47:57.016238       1 main.go:423] "serving encrypted healthz endpoint" path=":8443/healthz"
2023/03/08 14:48:33 [DEBUG] reviewing request 9e2625ea-29ab-4612-96bd-a1696ec664f6, named: default/
I0308 14:48:33.636791       1 main.go:142] "found pod to mutate" pod="default/"
I0308 14:48:33.636833       1 pod.go:285] "creating client certificate to use with auth service" default/="(MISSING)"
I0308 14:48:33.636852       1 clientCert.go:25] "creating x509 key pair for ca cert and key"
I0308 14:48:33.637003       1 clientCert.go:32] "parse certificate"
I0308 14:48:33.637038       1 clientCert.go:38] "generating client key"
I0308 14:48:33.687971       1 clientCert.go:44] "generating serial number"
I0308 14:48:33.687998       1 clientCert.go:66] "crating x509 certificate"
I0308 14:48:33.690387       1 pod.go:292] "mutate init-containers" default/="(MISSING)"
I0308 14:48:33.690427       1 pod.go:298] "mutate containers" default/="(MISSING)"
I0308 14:48:33.690434       1 pod.go:116] "found container to mutate" container="default/<<MyAplication>>"
I0308 14:48:33.690441       1 pod.go:119] "checking for env vars to inject" container="default/<<MyAplication>>"
I0308 14:48:33.690452       1 pod.go:122] "found env var to inject" env="<<MySecret>>@azurekeyvault" container="default/<<MyAplication>>"
I0308 14:48:33.690460       1 registry.go:30] "getting container command for container" container="default/<<MyAplication>>"
I0308 14:48:33.690469       1 registry.go:36] "no cmd override in kubernetes for container, checking docker image configuration for entrypoint and cmd" image="<<MyACR>>.azurecr.io/<<MyAplication>>:xxxxxxx4910f64cdc59xxxxxxxxxxxxxxx" container="default/<<MyAplication>>"
E0308 14:48:33.828678       1 main.go:165] "failed to mutate" err=<
        failed to get auto cmd, error: GET https://<<MyACR>>.azurecr.io/oauth2/token?scope=repository%3A<<MyAplication>>%3Apull&service=<<MyACR>>.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.
        cannot fetch image descriptor
        github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144
        github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(*Registry).GetImageConfig
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103
        main.getContainerCmd
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39
        main.podWebHook.mutateContainers
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143
        main.podWebHook.mutatePodSpec
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299
        main.vaultSecretsMutator
                /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163
        github.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate
                /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25
        github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview
                /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:128
        github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review
                /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:120
        github.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review
                /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/internal/instrumenting/instrumenting.go:42
        github.com/slok/kubewebhook/pkg/http.HandlerFor.func1
                /go/pkg/mod/github.com/slok/[email protected]/pkg/http/handler.go:64
        net/http.HandlerFunc.ServeHTTP
                /usr/local/go/src/net/http/server.go:2109
        github.com/gorilla/mux.(*Router).ServeHTTP
                /go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210
        net/http.serverHandler.ServeHTTP
                /usr/local/go/src/net/http/server.go:2947
        net/http.(*conn).serve
                /usr/local/go/src/net/http/server.go:1991
        runtime.goexit
                /usr/local/go/src/runtime/asm_amd64.s:1594
 > pod="default/"
2023/03/08 14:48:33 [ERROR] admission webhook error: failed to get auto cmd, error: GET https://<<MyACR>>.azurecr.io/oauth2/token?scope=repository%3A<<MyAplication>>%3Apull&service=<<MyACR>>.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.
cannot fetch image descriptor
github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144
github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(*Registry).GetImageConfig
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103
main.getContainerCmd
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39
main.podWebHook.mutateContainers
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143
main.podWebHook.mutatePodSpec
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299
main.vaultSecretsMutator
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163
github.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate
        /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25
github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview
        /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:128
github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review
        /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:120
github.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review
        /go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/internal/instrumenting/instrumenting.go:42
github.com/slok/kubewebhook/pkg/http.HandlerFor.func1
        /go/pkg/mod/github.com/slok/[email protected]/pkg/http/handler.go:64
net/http.HandlerFunc.ServeHTTP
        /usr/local/go/src/net/http/server.go:2109
github.com/gorilla/mux.(*Router).ServeHTTP
        /go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210
net/http.serverHandler.ServeHTTP
        /usr/local/go/src/net/http/server.go:2947
net/http.(*conn).serve
        /usr/local/go/src/net/http/server.go:1991
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1594

The message in kubectl describe replicaset <<MyApplicationRS>>:

Warning FailedCreate 54m replicaset-controller Error creating: Internal error occurred: failed calling webhook "pods.env-injector.admission.spv.no": failed to call webhook: an error on the server ("{\"response\":{\"uid\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"failed to get auto cmd, error: GET https://<<MyACR>>.azurecr.io/oauth2/token?scope=repository%3A<<MyApplication>>%3Apull\\u0026service=<<MyACR>>.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\\ncannot fetch image descriptor\\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144\\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(*Registry).GetImageConfig\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103\\nmain.getContainerCmd\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39\\nmain.podWebHook.mutateContainers\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143\\nmain.podWebHook.mutatePodSpec\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299\\nmain.vaultSecretsMutator\\n\\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:128\\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:120\\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review\\n\\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/in") has prevented the request from succeeding

I have reviewed if the Managed Identity showed in the injector pod has the image pull permissions into the acr and the permissions are assigned.

I0308 14:47:56.997902 1 provider.go:274] "azure: using managed identity extension to retrieve access token" id="xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxxxxx"

If I set the docker cmd as a command into kubernetes template, the image dont need be pulled, so env injection works correcly.

Is there something I can review?

Mar 08 '23 20:03 TomasGrinspon

Hello. I could reproduce the issue with a kubernetes cluster automatically created managed identity. I assigned that managed identity "AcrPull" on my azure container registry, but the env-injector was not able to pull the docker image with the same error message. The used version was: chart-versions 2.3.4 app-versions: 1.4.0

As soon as i downgraded to chart verions 2.2.2, app version 1.3.1 the docker pull in the env injector worked.

Apr 27 '23 15:04 iitsBergenol

Guys, any update here?

We must upgrade from 2.2.2, because we need the bug fix for https://github.com/SparebankenVest/public-helm-charts/issues/77 which is first deployed in 2.3.0

May 30 '23 17:05 MarkKharitonov

So for the upgrade from 2.2.2 to 2.3.0 we have found the following workaround:

Use HELM chart 2.3.0
Pass 1.3.1 explicitly in the env_injector.envImage.tag and env_injector.image.tag HELM values.

May 30 '23 19:05 MarkKharitonov

@MarkKharitonov - Tried that,still getting the error -

2023/06/09 04:26:06 [ERROR] admission webhook error: failed to get auto cmd, error: GET https://myprivateacr.azurecr.io/oauth2/token?scope=repository%3Aclient%3Apull&service=myprivateacr.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.
akv2k8s-envinjector-6865d94856-pfvlg webhook cannot fetch image descriptor
akv2k8s-envinjector-6865d94856-pfvlg webhook github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144
akv2k8s-envinjector-6865d94856-pfvlg webhook github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(*Registry).GetImageConfig
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103
akv2k8s-envinjector-6865d94856-pfvlg webhook main.getContainerCmd
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39
akv2k8s-envinjector-6865d94856-pfvlg webhook main.podWebHook.mutateContainers
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143
akv2k8s-envinjector-6865d94856-pfvlg webhook main.podWebHook.mutatePodSpec
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299
akv2k8s-envinjector-6865d94856-pfvlg webhook main.vaultSecretsMutator
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163
akv2k8s-envinjector-6865d94856-pfvlg webhook github.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25
akv2k8s-envinjector-6865d94856-pfvlg webhook github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:128
akv2k8s-envinjector-6865d94856-pfvlg webhook github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:120
akv2k8s-envinjector-6865d94856-pfvlg webhook github.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/internal/instrumenting/instrumenting.go:42
akv2k8s-envinjector-6865d94856-pfvlg webhook github.com/slok/kubewebhook/pkg/http.HandlerFor.func1
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/pkg/mod/github.com/slok/[email protected]/pkg/http/handler.go:64
akv2k8s-envinjector-6865d94856-pfvlg webhook net/http.HandlerFunc.ServeHTTP
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/usr/local/go/src/net/http/server.go:2084
akv2k8s-envinjector-6865d94856-pfvlg webhook github.com/gorilla/mux.(*Router).ServeHTTP
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210
akv2k8s-envinjector-6865d94856-pfvlg webhook net/http.serverHandler.ServeHTTP
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/usr/local/go/src/net/http/server.go:2916
akv2k8s-envinjector-6865d94856-pfvlg webhook net/http.(*conn).serve
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/usr/local/go/src/net/http/server.go:1966
akv2k8s-envinjector-6865d94856-pfvlg webhook runtime.goexit
akv2k8s-envinjector-6865d94856-pfvlg webhook 	/usr/local/go/src/runtime/asm_amd64.s:1571

And below are the helm values -

    new k8s.helm.v3.Chart(
      "akv2k8s",
      {
        chart: "akv2k8s",
        version: "2.3.0", 
        namespace: platformToolsNamespace.id,
        fetchOpts: {
          repo: "https://charts.spvapi.no",
        },
        values: {
          controller: {
            tolerations: [
              {
                key: "kubernetes.azure.com/scalesetpriority",
                operator: "Exists",
              },
            ],
          },
          env_injector: {
            envImage: {
              tag: "1.3.1"
            },
            image: {
              tag: "1.3.1"
            },
            authService: false,
            tolerations: [
              {
                key: "kubernetes.azure.com/scalesetpriority",
                operator: "Exists",
              },
            ],
          },
        },
      },
      { provider: this.kubeProvider }
    );

Anything else to try out. Really need this fixed.

Jun 09 '23 04:06 ahilmathew

I do not know what to add. We currently deploy the helm chart using the terraform helm_release resource:

resource "helm_release" "akv2k8s" {
  name       = "akv2k8s"
  chart      = "akv2k8s"
  version    = "2.3.0"
  namespace  = kubernetes_namespace.akv2k8s.metadata[0].name
  repository = "http://charts.spvapi.no"
  set {
    name  = "addAzurePodIdentityException"
    value = local.addAzurePodIdentityException
  }
  set {
    name  = "env_injector.envImage.tag"
    value = "1.3.1"
  }
  set {
    name  = "env_injector.image.tag"
    value = "1.3.1"
  }
}

addAzurePodIdentityException is likely irrelevant to you.

Jun 09 '23 13:06 MarkKharitonov

I am using pod identity as well. I have kept authService: false, as my assumption is that akv2k8s will use the identity assigned to the pod.

Jun 10 '23 11:06 ahilmathew

we're fighting the same issue..... I noticed this: https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/releases/tag/webhook-1.4.1

in the mean time, we pinned our image tag versions to 1.3.1 (as described by @MarkKharitonov) and testing seems to work

Jun 21 '23 15:06 jasonrberk

Same issue, using chart 2.4.2, with spvest/azure-keyvault-webhook:1.4.0 gives the error while 1.3.1 works fine. Using aks system identity for everything.

Jun 28 '23 16:06 daniel-anova

Hi all,

I'd like to address the "elephant in the room" -- AAD Pod Identity is no longer supported by Microsoft, and instead it is recommended to use AZWI. That being said, I've been able to get AKV2K8S working with a custom UMI without even AZWI. Note I did not try using kubelet identity, as I have a security requirement to use a separate identity from the kubelet in my org, but neither AZWI nor AADPI is required with env-injector v1.4.1.

My values:

  global:
    keyVaultAuth: azureCloudConfig
    userDefinedMSI:
      enabled: true
      msi: "<my-identity-client-id>"
      subscriptionId: "<my-subscription-id>"
      tenantId: "<my-tenant-id>"
      azureCloudType: "azurePublicCloud"
    metrics:
      enabled: true

The only other steps I needed to take was to add the UMI associated with <my-identity-client-id> to the VMSS in the portal and enable the cluster OIDC issuer. I should note here that I'm not using ACR, but if you are and you want to try the custom UMI option, then definitely grant the custom UMI the AcrPull permission). I do not have AZWI nor AADPI installed in my cluster.

For those who want to use kubelet identity and already granted the kubelet identity the AcrPull privileges, try the chart 2.4.2 with env-injector 1.4.1 and the values above -- replace msi value with the client ID of your kubelet identity. Also if you only installed AADPI for AKV2K8S, you can uninstall it with this setup. I'm not certain it will work, but it will provide additional data to help potentially narrow the cause of the problems you're having (especially if this does work)

Jun 30 '23 14:06 tspearconquest

azure-key-vault-to-kubernetes azure-key-vault-to-kubernetes copied to clipboard

[BUG] envinjector unable to pull docker image from azurecr using aadpodidbinding

azure-key-vault-to-kubernetes
azure-key-vault-to-kubernetes copied to clipboard