[argocd-vault-plugin] Error in fetching secret from Azure

[om3171991]

Describe the bug While trying to fetch secret from Azure vault using argocd-vault-plugin, we are facing below error message

keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

cmd run to get secret on repo server pod: cat example.yaml |argocd-vault-plugin -c vault.yaml generate - Output : keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded example.yaml kind: Secret apiVersion: v1 metadata: name: example-annotation annotations: avp.kubernetes.io/path: "azure-vault" type: Opaque data: username: <USER-SECRET>

cat vault.yaml AZURE_CLIENT_SECRET: <Removed> AZURE_CLIENT_ID: <Removed> AZURE_TENANT_ID: <Removed> AVP_TYPE: azurekeyvault

Expected behavior The secret value should be fetched from the Azure vault

Additional context We have looked Azure portal for a similar issue and tried curl from repo-server as asked (https://docs.microsoft.com/en-us/azure/aks/csi-secrets-store-troubleshooting#keyvaultbaseclientgetsecret-failure-sending-request-statuscode0--original-error-context-canceled), I see values are coming from the vault as expected.

Please help in debugging.

[jkayani]

Hi,

I looked at the link you gave which specified these instructions:

curl -X POST 'https://login.microsoftonline.com/<AAD_TENANT_ID>/oauth2/v2.0/token' -d 'grant_type=client_credentials&client_id=<AZURE_CLIENT_ID>&client_secret=<AZURE_CLIENT_SECRET>&scope=https://vault.azure.net/.default'

followed by

curl -X GET 'https://<KEY_VAULT_NAME>.vault.azure.net/secrets/<SECRET_NAME>?api-version=7.2' -H "Authorization: Bearer <ACCESS_TOKEN_ACQUIRED_ABOVE>"

Can you verify that the name of your KeyVault instance is azure-vault? By setting the avp.kubernetes.io/path annotation to that value, you're telling AVP your KeyVault instance is located at the URL: azure-vault.vault.azure.net: https://github.com/argoproj-labs/argocd-vault-plugin/blob/main/pkg/backends/azurekeyvault.go#L32.

If that value is correct, maybe your instance is taking longer than 10 seconds to respond with the secrets? https://github.com/argoproj-labs/argocd-vault-plugin/blob/main/pkg/backends/azurekeyvault.go#L34. You can try timing the response in your curl test to see, or try increasing the timeout in the AVP code and then recompile and load into the repo-server and testing that way. Not sure how likely this is to be the problem.

Otherwise, I'll have to look deeper (and could use your eyes on the code here: https://github.com/argoproj-labs/argocd-vault-plugin/blob/main/pkg/backends/azurekeyvault.go#L1 since I don't use the Azure KeyVault product myself).

[om3171991]

Hi @jkayani,

I double-checked on timeouts and with MSFT curls, we are getting responses in a max of 2 seconds and the name of the vault is correct. Don't have much idea of Go but will try to recompile code with a higher timeout and request if can also lookout.

[loxley]

Hey @om3171991 do you have more than 25 secrets in your vault?

I looked a bit at the code and this line, where it says nil is an option that can be set to how many secrets that maximum should be fetched:

https://github.com/argoproj-labs/argocd-vault-plugin/blob/b434368a10e088add8c47e0a8be0a003376436a8/pkg/backends/azurekeyvault.go#L39

Docs for the final function GetSecrets:

// GetSecrets the Get Secrets operation is applicable to the entire vault. However, only the base secret identifier and
// its attributes are provided in the response. Individual secret versions are not listed in the response. This
// operation requires the secrets/list permission.
// Parameters:
// vaultBaseURL - the vault name, for example https://myvault.vault.azure.net.
// maxresults - maximum number of results to return in a page. If not specified, the service will return up to
// 25 results.

I dont use Azure key vault anymore but setting that to a higher number might fix it. Please tell if you have more than 25 secrets.

[werne2j]

@om3171991 are you still having issues?

[avdicl]

I'm having this issue from an ArgoCD plugin sidecar. I'm migrating from argocd-cm plugins to sidecar plugins and all of my sidecars that require argocd-vault-plugin have this issue.

[werne2j]

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/troubleshoot-key-vault-csi-secrets-store-csi-driver#cause-2-keyvaultbaseclientgetsecret-failure-sending-request-statuscode0-error This issue is related to the repo server not able to access key vault. Hopefully the link will help. I would check for networking policies on your pods and see if something is preventing that.