Unable to pull images from Azure container registry when using workload identity and akv2k8s together. We have updated the chart version to 2.3.1 and made necessary changes to CRDs as per #442

[kartheek-cervello]

Note: Make sure to check out known issues (https://github.com/sparebankenvest/azure-key-vault-to-kubernetes#known-issues) before submitting

Your question Unable to pull images from Azure container registry when using workload identity and akv2k8s together. We have updated the chart version to 2.3.1 and made necessary changes to CRDs as per #442

To Reproduce helm chart configuration for akv2k8s controller.image: tag: 1.5.0-beta.1 controller.podLabels: | "azure.workload.identity/use": "true" controller.serviceAccount.annotations: | "azure.workload.identity/client-id": "xxxxxxxxx" controller.serviceAccount.labels: | "azure.workload.identity/use": "true" env_injector.envImage.tag: 1.5.0-beta.1 env_injector.image.tag: 1.5.0-beta.1 env_injector.podLabels: | "azure.workload.identity/use": "true" env_injector.serviceAccount.annotations: | "azure.workload.identity/client-id": "xxxxxxx" env_injector.serviceAccount.labels: | "azure.workload.identity/use": "true" global.keyVaultAuth: environment-azidentity

manifest file of the POD

deploy a pod that references the service account

apiVersion: v1 kind: Pod metadata: name: quick-start namespace: sndbx-2-eus2 spec: serviceAccountName: ksf-sndbx-2-eus2-agentpool containers: - image: xxx.azurecr.io/xxx/dev/app:latest](http://xxx.azurecr.io/xxx/dev/app:latest name: oidc env: - name: KEYVAULT_NAME value: xxxxxxxx - name: SECRET_NAME value: "secret-name@azurekeyvault" - name: KEYVAULT_URL value: https://xxxxxxxxx.vault.azure.net/

kvmap manifest file apiVersion: spv.no/v2beta1 kind: AzureKeyVaultSecret metadata: name: secret-name namespace: sndbx-2-eus2 spec: vault: name: xxxxxxxxx #Key Vault Name object: name: SECRET-NAME # Name of Secret type: secret # Type of object: key, secret, certificate

Logs Error from server (InternalError): error when creating "demo.yaml": Internal error occurred: failed calling webhook "pods.env-injector.admission.spv.no": failed to call webhook: an error on the server ("{"response":{"uid":"80361cd7-cf79-41f8-acaa-b20405c3890b","allowed":false,"status":{"metadata":{},"status":"Failure","message":"failed to get auto cmd, error: GET https://xxx.azurecr.io/oauth2/token?scope=repository%3Apact%2Fdev%2Fapp%3Apull\u0026service=xxx.azurecr.io: UNAUTHORIZED: authentication required, visit https://aka.ms/acr/authorization for more information.\ncannot fetch image descriptor\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getImageConfig\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:144\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(*Registry).GetImageConfig\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:103\nmain.getContainerCmd\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39\nmain.podWebHook.mutateContainers\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:143\nmain.podWebHook.mutatePodSpec\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:299\nmain.vaultSecretsMutator\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:163\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/mutator.go:25\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:128\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/mutating/webhook.go:120\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review\n\t/go/pkg/mod/github.com/slok/[email protected]/pkg/webhook/internal/inst") has prevented the request from succeeding

[cgroschupp]

I have tested the env_injector with the workload identity support and I do not have the problem.

UNAUTHORIZED: authentication required do you have access to the image? Maybe the imagePullSecret or permission is missing?

@kartheek-cervello on azure you can use az aks check-acr to check the access.

My helm values:

controller:
  image:
    tag: 1.5.0-beta.1
  logLevel: info
  podLabels:
    azure.workload.identity/use: "true"
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: <myid>
    create: true
    labels:
      azure.workload.identity/use: "true"
env_injector:
  enabled: true
  envImage:
    tag: 1.5.0-beta.1
  image:
    tag: 1.5.0-beta.1
  podLabels:
    azure.workload.identity/use: "true"
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: <myid>
    create: true
    labels:
      azure.workload.identity/use: "true"
    name: akv2k8s-envinjector
global:
  keyVaultAuth: environment-azidentity

My manifests:

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
   - image: mytest.azurecr.io/alpine:3.17
     imagePullPolicy: Always
     command:
     - env
     name: test
     env:
     - name: SECRET_NAME
       value: "test-username-inject@azurekeyvault"
---
apiVersion: spv.no/v1
kind: AzureKeyVaultSecret
metadata:
  name: test-username-inject
  namespace: test
spec:
  vault:
    name: mykeyvault
    object:
      name: test-username
      type: secret

Pod output:

test I0202 12:08:05.597938       1 version.go:31] "version info" version="" commit="79d3e52" buildDate="2022-12-28T20:31:03Z" component="vaultenv"
test I0202 12:08:05.598029       1 main.go:185] "azure key vault env injector initializing"
test I0202 12:08:05.598209       1 main.go:254] "found original container command" cmd="/usr/bin/env" args=[env]
test I0202 12:08:05.598262       1 authentication.go:110] "checking if current auth service credentials are stale" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/test/test?secret=akv2k8s-test"
test I0202 12:08:05.680814       1 authentication.go:123] "auth service credentials ok" url="http://akv2k8s-envinjector.akv2k8s.svc:80/auth/test/test?secret=akv2k8s-test"
test I0202 12:08:05.681355       1 authentication.go:159] "requesting azure key vault oauth token" url="https://akv2k8s-envinjector.akv2k8s.svc:9443/auth/test/test"
test I0202 12:08:05.707612       1 authentication.go:179] "successfully received oauth token"
test I0202 12:08:05.846204       1 main.go:353] "secret injected into env var" azurekeyvaultsecret="test/test-username-inject" env="SECRET_NAME"
test I0202 12:08:05.846280       1 main.go:359] "starting process with secrets in env vars" cmd="/usr/bin/env" args=[env]
...
test SECRET_NAME=myusername
...
Stream closed EOF for test/test (test)
Stream closed EOF for test/test (copy-azurekeyvault-env)

[181192]

@kartheek-cervello @cgroschupp We have a bug indeed... your example does not trigger this scenario. It only triggers when the container command is not set, when we need to query the container registry to get the ENTRYPOINT or CMD from the manifest https://github.com/SparebankenVest/azure-key-vault-to-kubernetes/blob/master/cmd/azure-keyvault-secrets-webhook/registry.go#L35

[prestonr83]

Can confirm I am experiencing the same issue using chart 2.3.1 on a clean install. Downgrading to 2.2.2 and it works fine.

[malvegil]

Also confirmed resolution with version 2.2.2 of the chart.

[wsmelton]

@cgroschupp reference your comment you are using a beta release version of the image.

Are users/environments expected to use a pre-release version in production environments to make this work again?

[cgroschupp]

@wsmelton The Workload Identity feature is currently not released, if you want to test it you will need to use the pre-release.

@kartheek-cervello Can you try what I mentioned here?

[mozts2005]

@cgroschupp and @181192 now that Workload Identity is GA Is there an ETA for a release?