ErrAzureVault: Failed to get secret from Azure Key Vault

[ukreddy-erwin]

I am running from a user through which test environment was created on aks. I followed the steps mentioned in the quick start guide and below is the error I got when I describe the secret CRD.

  Type     Reason         Age                     From                     Message
  ----     ------         ----                    ----                     -------
  Warning  ErrAzureVault  7m12s (x13 over 8m12s)  azurekeyvaultcontroller  Failed to get secret for 'secret-sync' from Azure Key Vault 'akv2k8s-test'
  Warning  ErrAzureVault  4m12s (x31 over 9m12s)  azurekeyvaultcontroller  Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'

More detailed:

PS C:\Users\UReddy\akv2k8s> kubectl get azurekeyvaultsecret.spv.no/secret-sync -n akv-test
NAME          VAULT           VAULT OBJECT   SECRET NAME   SYNCHED
secret-sync   testingvaultd   my-secret
PS C:\Users\UReddy\akv2k8s> kubectl describe azurekeyvaultsecret.spv.no/secret-sync -n akv-test
Name:         secret-sync
Namespace:    akv-test
Labels:       <none>
Annotations:  <none>
API Version:  spv.no/v1
Kind:         AzureKeyVaultSecret
Metadata:
  Creation Timestamp:  2021-08-05T12:38:59Z
  Generation:          3
  Managed Fields:
    API Version:  spv.no/v2beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:output:
          .:
          f:secret:
            .:
            f:dataKey:
            f:name:
        f:vault:
          .:
          f:name:
          f:object:
            .:
            f:name:
            f:type:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-08-05T12:38:59Z
  Resource Version:  12022
  UID:               007f2f4c-5d59-43ee-a95d-625af8b3aee8
Spec:
  Output:
    Secret:
      Data Key:  secret-value
      Name:      my-secret-from-akv
  Vault:
    Name:  testingvaultd
    Object:
      Name:  my-secret
      Type:  secret
Events:
  Type     Reason         Age                     From                     Message
  ----     ------         ----                    ----                     -------
  Warning  ErrAzureVault  7m12s (x13 over 8m12s)  azurekeyvaultcontroller  Failed to get secret for 'secret-sync' from Azure Key Vault 'akv2k8s-test'
  Warning  ErrAzureVault  4m12s (x31 over 9m12s)  azurekeyvaultcontroller  Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
PS C:\Users\UReddy\akv2k8s>

Please suggest how to fix that?

[kristeey]

Please check the Controller logs to obtain more information why it fails (https://akv2k8s.io/troubleshooting/controller-log/). This warning is often occur when akv2k8s do not have Get permission on the keyvault. Ensure that you have added an access policy for the keyvault giving access to the service principal or managed identity that you use.

[kristeey]

@ukreddy-erwin Did you find a solution to your problem?

[ukreddy-erwin]

The controller log is showing like this.

 'secret-sync' from Azure Key Vault 'testingvaultd'
E0806 12:09:54.954047       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0806 12:09:54.954353       1 worker.go:106] Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'

But,as in the getting started page, "By default akv2k8s use the AKS cluster credentials to authenticate with Azure Key Vault". Why it is not picking then? Both vault and aks are created under same account and Resource group also

[kristeey]

Creating an AKS cluster requires an identity of some sort to manage cloud resources attached to the cluster (such as loadbalancers, managed disks etc.). This identity can either be (i) a Managed Identity (identity of an application) or (ii) a Service Principal. A Managed Identity will be created automatically for your cluster upon installation if you do not specify otherwise. If you choose to use a service principal, you need to provide one, or AKS will create one on your behalf.

"By default akv2k8s use the AKS cluster credentials to authenticate with Azure Key Vault" means that akv2k8s tool will use this identity/principal that was created along side your cluster, by default, to authenticate with the Azure Key Vault. Hence, in order to give access to the Azure Key Vault you need to update the Access Policy of that Key Vault to include GET permission given to the identity/principal.

You can find the ID of the managed identity you use with this command az aks show -g <RGName> -n <ClusterName> --query "servicePrincipalProfile"

If you wish custom authentication you can do that by setting controller.keyVaultAuth=environment in the helm installation.

[ukreddy-erwin]

I created Azure with Managed Identity,system defined one. And using the same account subscription, I created azure vault in the same resource group.

And the vault permission model is Vault access policy

I have provided full access to my user account in vault with all options related to keys,secrets and certificates selected.

But still getting this error.

[kristeey]

It does not matter if your user account has access to the vault. What matters is that the managed identity (used by AKS) has access to the vault. Can you confirm that the managed identity (that is listed when you runaz aks show -g <RGName> -n <ClusterName> --query "servicePrincipalProfile") has Get Secrets permission on the vault by doing the following:

1. Open the portal
2. Navigate to the keyvault you want Akv2k8s to get secrets from (testingvaultd)
3. Go to Settings->Access Policies
4. Ensure that the managed identity has Get Secrets permission (assuming it is secrets you want to sync). If the managed identity do not have this permission, click Add Access Policy -> choose Get Secret Permission and enter the clientID (from the az aks show ... command) in the Select principal section.

Btw, The Controller logs that you provided is showing the "clean" version of the original (more detailed) error log entry which you will find if you scroll up to where the error starts. In the more detailed error log you should find additional information on why it fails getting secrets from the keyvault. After a while the detailed logging is switched with a "cleaned up" version of the error log to avoid to much noise in the log.

[ukreddy-erwin]

From my azure command line, I got below output.

az aks show -g test -n aksvault --query "servicePrincipalProfile"
{
  "clientId": "msi"
}

But the name is different in the access policies section of azure vault. It shows my full name and my company email ID here.

[kristeey]

Okey, as suspected there is no access policy for the Managed identity. Just click + Add Access Policy and add the msi.

Remember to save afterwards 👍

[ukreddy-erwin]

It is giving only option to add organization members to it. There is no option for msi.

[kristeey]

The ClientID should be a GUID (combination of numbers and letters) and not just "msi". Just to confirm, is the name of the Kubernetes cluster aksvault and the resource group name where the test?

[ukreddy-erwin]

yes the k8s clustername is aksvault resource group is test and in the same resource group testingvaultd azure vault is created. And I guess it should be the first option only for permission model.

Permission model Vault access policy Azure role-based access control

[ukreddy-erwin]

I added below apps also to vault policies.

which I found here.

But same issue

[ukreddy-erwin]

As it is managed identity, that's why it is showing as "msi" when ran the command

az aks show -g test -n aksvault --query "servicePrincipalProfile" { "clientId": "msi" }

https://docs.microsoft.com/en-us/azure/aks/use-managed-identity#code-try-6

az aks show -g test -n aksvault --query "identity" { "principalId": "", "tenantId": "****", "type": "SystemAssigned", "userAssignedIdentities": null }

If I try to use that principal ID in the above output to the policy assign command, I am getting error as below.

az keyvault set-policy -n testingvaultd --secret-permissions get --spn "caf1a7d6-57a0-4208-b46d-7f299d69c07d"

Unable to find user with spn 'caf1a7d6-57a0-4208-b46d-7f299d69c07d' Unable to get object id from principal name.

[kristeey]

Could you see if there is an application named aksvault in AAD. Follow these steps:

1. Go to Azure Active Directory -> Enterprise Applications
2. Choose All Applications in the Application type drop down menu.
3. Search for aksvault. There should be an application with the same name as the cluster. Add that application to the access policies

[ukreddy-erwin]

I ran below command

az keyvault set-policy -n testingvaultd --secret-permissions get --spn 763f5805-cea9-483c-9b02-8b55f3106ac9

from here app id i copied

But still it is not showing the secret

kubectl get azurekeyvaultsecret.spv.no/secret-sync -n akv-test
NAME          VAULT           VAULT OBJECT   SECRET NAME   SYNCHED
secret-sync   testingvaultd   my-secret

event the event is still as below

Events:
  Type     Reason         Age                      From                     Message
  ----     ------         ----                     ----                     -------
  Warning  ErrAzureVault  40s (x68857 over 3d23h)  azurekeyvaultcontroller  Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'

but there is new policy added.

[kristeey]

Could you provide the full Controller log removing the repeting stuff?

[ukreddy-erwin]

these lines are repeated in the logs

E0809 12:23:54.859707       1 worker.go:106] Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 12:24:23.091540       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 12:24:23.586808       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 12:24:23.902571       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 12:24:24.214305       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 12:24:24.730111       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 12:24:25.119606       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 12:24:25.119736       1 worker.go:106] Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'

[kristeey]

That is not the full controller log

[ukreddy-erwin]

can you check this

https://raw.githubusercontent.com/uday1kiran/logs/master/aks/controller.log

kubectl -n akv2k8s logs deployment/akv2k8s-controller > controller.log

[kristeey]

Okey, thanks. I see. The pod has been running for so long that we can only see the latest log entries. If you delete the pod, a new pod will be created and we can get a fresh log including the start-up log. Please delete the pod kubectl -n akv2k8s delete pod <controller-pod-name> and provide the log from the new pod using kubectl -n akv2k8s logs <controller-pod-name>. Wait for the controller to fail before sending the logs again.

[ukreddy-erwin]

same error now also.

PS C:\Users\UReddy> kubectl -n akv2k8s logs akv2k8s-controller-d7c8c5f78-rndpb
I0809 13:06:25.112138       1 main.go:92] "log settings" format="text" level="2"
I0809 13:06:25.112191       1 version.go:31] "version info" version="1.2.3" commit="bb751ae" buildDate="2021-03-25T09:49:14Z" component="controller"
W0809 13:06:25.112483       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0809 13:06:25.114732       1 main.go:129] "Creating event broadcaster"
I0809 13:06:25.128986       1 controller.go:167] "setting up event handlers"
I0809 13:06:25.129027       1 controller.go:178] "starting azurekeyvaultsecret controller"
I0809 13:06:25.129136       1 reflector.go:219] Starting reflector *v2beta1.AzureKeyVaultSecret (30s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
I0809 13:06:25.129488       1 reflector.go:219] Starting reflector *v1.ConfigMap (30s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
I0809 13:06:25.129690       1 reflector.go:219] Starting reflector *v1.Secret (30s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167
I0809 13:06:25.229283       1 controller.go:196] "starting azure key vault secret queue"
I0809 13:06:25.229556       1 controller.go:199] "starting azure key vault deleted secret queue"
I0809 13:06:25.229637       1 controller.go:202] "starting azure key vault queue"
I0809 13:06:25.229786       1 controller.go:205] "started workers"
E0809 13:06:25.574449       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: failed to get secret from Azure Key Vault for secret 'akv-test'/'secret-sync', error: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The policy requires the caller 'appid=f969f52b-c6a6-41c7-b6ef-949c1f227b1d;oid=66d10cbf-3a73-4326-9a8d-b1b082e7c3fc;iss=https://sts.windows.net/f33abe27-86cd-46d6-ae2b-b918362ab160/' to use on-behalf-of (OBO) flow. For more information on OBO, please see https://go.microsoft.com/fwlink/?linkid=2152310" InnerError={"code":"ForbiddenByPolicy"}
E0809 13:06:25.876043       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: failed to get secret from Azure Key Vault for secret 'akv-test'/'secret-sync', error: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The policy requires the caller 'appid=f969f52b-c6a6-41c7-b6ef-949c1f227b1d;oid=66d10cbf-3a73-4326-9a8d-b1b082e7c3fc;iss=https://sts.windows.net/f33abe27-86cd-46d6-ae2b-b918362ab160/' to use on-behalf-of (OBO) flow. For more information on OBO, please see https://go.microsoft.com/fwlink/?linkid=2152310" InnerError={"code":"ForbiddenByPolicy"}
E0809 13:06:26.208825       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: failed to get secret from Azure Key Vault for secret 'akv-test'/'secret-sync', error: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The policy requires the caller 'appid=f969f52b-c6a6-41c7-b6ef-949c1f227b1d;oid=66d10cbf-3a73-4326-9a8d-b1b082e7c3fc;iss=https://sts.windows.net/f33abe27-86cd-46d6-ae2b-b918362ab160/' to use on-behalf-of (OBO) flow. For more information on OBO, please see https://go.microsoft.com/fwlink/?linkid=2152310" InnerError={"code":"ForbiddenByPolicy"}
E0809 13:06:26.517032       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: failed to get secret from Azure Key Vault for secret 'akv-test'/'secret-sync', error: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The policy requires the caller 'appid=f969f52b-c6a6-41c7-b6ef-949c1f227b1d;oid=66d10cbf-3a73-4326-9a8d-b1b082e7c3fc;iss=https://sts.windows.net/f33abe27-86cd-46d6-ae2b-b918362ab160/' to use on-behalf-of (OBO) flow. For more information on OBO, please see https://go.microsoft.com/fwlink/?linkid=2152310" InnerError={"code":"ForbiddenByPolicy"}
E0809 13:06:26.895845       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: failed to get secret from Azure Key Vault for secret 'akv-test'/'secret-sync', error: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The policy requires the caller 'appid=f969f52b-c6a6-41c7-b6ef-949c1f227b1d;oid=66d10cbf-3a73-4326-9a8d-b1b082e7c3fc;iss=https://sts.windows.net/f33abe27-86cd-46d6-ae2b-b918362ab160/' to use on-behalf-of (OBO) flow. For more information on OBO, please see https://go.microsoft.com/fwlink/?linkid=2152310" InnerError={"code":"ForbiddenByPolicy"}
E0809 13:06:27.265638       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: failed to get secret from Azure Key Vault for secret 'akv-test'/'secret-sync', error: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The policy requires the caller 'appid=f969f52b-c6a6-41c7-b6ef-949c1f227b1d;oid=66d10cbf-3a73-4326-9a8d-b1b082e7c3fc;iss=https://sts.windows.net/f33abe27-86cd-46d6-ae2b-b918362ab160/' to use on-behalf-of (OBO) flow. For more information on OBO, please see https://go.microsoft.com/fwlink/?linkid=2152310" InnerError={"code":"ForbiddenByPolicy"}
E0809 13:06:27.265664       1 worker.go:106] failed to get secret from Azure Key Vault for secret 'akv-test'/'secret-sync', error: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The policy requires the caller 'appid=f969f52b-c6a6-41c7-b6ef-949c1f227b1d;oid=66d10cbf-3a73-4326-9a8d-b1b082e7c3fc;iss=https://sts.windows.net/f33abe27-86cd-46d6-ae2b-b918362ab160/' to use on-behalf-of (OBO) flow. For more information on OBO, please see https://go.microsoft.com/fwlink/?linkid=2152310" InnerError={"code":"ForbiddenByPolicy"}
E0809 13:06:55.552180       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 13:06:55.945959       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 13:06:56.300826       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 13:06:56.613631       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 13:06:56.960967       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 13:06:57.330274       1 worker.go:92] Failed to process key akv-test/secret-sync. Reason: Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'
E0809 13:06:57.330907       1 worker.go:106] Failed to get secret for 'secret-sync' from Azure Key Vault 'testingvaultd'

[kristeey]

So this is the error:

failed to get secret from Azure Key Vault for secret 'akv-test'/'secret-sync', error: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The policy requires the caller 'appid=f969f52b-c6a6-41c7-b6ef-949c1f227b1d;oid=66d10cbf-3a73-4326-9a8d-b1b082e7c3fc;iss=https://sts.windows.net/f33abe27-86cd-46d6-ae2b-b918362ab160/' to use on-behalf-of (OBO) flow. For more information on OBO, please see https://go.microsoft.com/fwlink/?linkid=2152310" InnerError={"code":"ForbiddenByPolicy"}

which indicate that the error is Policy related. Please check out https://docs.microsoft.com/en-us/answers/questions/117610/how-to-fix-34the-policy-requires-the-caller-3939-t.html

[ukreddy-erwin]

As per this link you shared https://docs.microsoft.com/en-us/answers/questions/117610/how-to-fix-34the-policy-requires-the-caller-3939-t.html it says to add the policy to application.

Which we already did earlier.

But still same issue

[jacobdanovitch]

Took a while to figure out but it's straightforward (even if not at all obvious):

# fill these in
KEYVAULT=
AKS=
AKS_RG=

az keyvault set-policy \
  -n $KEYVAULT \
  --secret-permissions get \
  --object-id $(az aks show --name $AKS -g $AKS_RG --query identityProfile.kubeletidentity.objectId)

If you're using a managed identity then it's the objectId of the kubelet identity instead of the SPN, which I think corresponds to the AKS agent pool. I believe this is what --attach-acr does in the background when creating the cluster as well, adding the AcrPull role onto the kubeletidentity's obectid (not sure though).

That was the only thing required for me, it worked immediately thereafter. Hope that helps.

[kristeey]

Does this work for you @ukreddy-erwin ? This makes sense as it is the objectId of the system assigned managed identity (with name <cluster-name>-agentpool) in the MC_... resource group created by aks.

[Bizzaro]

Hi, if anyone is still struggling with this, I used a managed identity. However, when you're giving access to the vault you have to select the nodepool , not the cluster itself.

Nowhere is this documented anywhere.

[cramakan]

I am also getting same error in AKS cluster with managed identity ,I added managed identity in key vault IAM as contributor permission also in access policy added permission as het list buts still no luck ,same is working with another AKS cluster with SPN, but somehow its not working with Managed identity.

[gldraphael]

Debugging tips for anyone else seeing this:

# check controller logs 
kubectl logs akv2k8s-controller-xxxxxxxxxx-xxxxx -n akv2k8s

Look for a line like:

Failed to process key namespacename/secretname. Reason: failed to get secret from Azure Key Vault for secret 'namespacename'/'secretname', error: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The user, group or application 'appid={GUID};oid={GUID};iss=https://sts.windows.net/{GUID}/' does not have secrets get permission on key vault 'vault-name;location=eastus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287" InnerError={"code":"AccessDenied"}

The appId value from this log line is the kubelet identity. Copy it and use search for it in your Active Directory. In most cases this should be {clusterName}-agentpool.

For more information on AKS identities see: Use Managed Identity.

[thisurawijesekera]

If you are using managed identity, you can get the object id of that using following command az aks show -n -g | jq .identityProfile.kubeletidentity.objectId -r

And you can set the policy for that object id that you recieved from previous command using Azure CLI like below or you can do that from the Azure UI.

az keyvault set-policy --name <KeyVault> --object-id <objectId> --certificate-permissions get

If you are using multline secret make sure to set the correct content type in AzureKeyVaultSecret eg: contentType: application/x-json or application/x-yaml If using multline secret I prefer to save as json in azure key vault and contentType: application/x-json. (I tried to save as yaml but it did not work)

[tspearconquest]

Voting to close as solution has been provided.

To be clear, when using SystemAssigned Managed Identity, at least 2 identities are created.

• {clustername} - SystemAssigned Managed Identity - this is assigned to the control-plane nodes which we can't see/access in Azure. We can only find this identity by searching it in AzureAD.
• {clustername}-agentpool - UserAssigned Managed Identity - this is assigned to the VMSS in the cluster management resource group (resource group name starting with MC_). This is the kubelet managed identity. It needs AcrPull privileges on the ACR registry (if using ACR), and at least Get and List on the Keyvault Access Policy.

The second one is the one. Copy that MI client ID and add it to your KV access policy as documented here and here