Kubernetes auth method does not work with "Use client JWT as reviewer JWT" option - with workaround

[jtv8]

Describe the bug One of the options for configuring the Kubernetes auth method described in Kubernetes auth method is "Use client JWT as reviewer JWT". This does not work out-of-the-box with VSO and will result in a HTTP 403 error for PUT https://{VAULT_ADDR}/v1/auth/{MOUNT_PATH} . This is because Vault attempts to authenticate to the Kubernetes TokenReview API with the client JWT, but this JWT is missing the appropriate audience claim for the Kubernetes APIServer.

To Reproduce Steps to reproduce the behavior:

1. Configure the Kubernetes Auth backend according to the supplied Terraform example
2. Deploy VSO using the supplied values.yaml
3. Apply the supplied CustomResources
4. Run kubectl describe VaultDynamicSecret app-db-creds -n demo-ns to see the following error:

Events:
  Type     Reason                  Age              From                Message
  ----     ------                  ----             ----                -------
  Warning  VaultClientConfigError  1s (x7 over 1s)  VaultDynamicSecret  Failed to get Vault client: Error making API request.

URL: PUT https://{VAULT_ADDR}/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied, lease_id=

Application deployment:

Auth config in Terraform:

resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
  path = "kubernetes"
}

resource "vault_kubernetes_auth_backend_config" "kubernetes_auth" {
  backend         = vault_auth_backend.kubernetes.path
  kubernetes_host = one(data.azurerm_kubernetes_cluster.aks.kube_config).host
  kubernetes_ca_cert = base64decode(
    one(data.azurerm_kubernetes_cluster.aks.kube_config).cluster_ca_certificate
  )
  disable_local_ca_jwt = true
}

values.yaml

defaultVaultConnection:
  enabled: true
  address: https://{VAULT_ADDR}

CustomResources:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: app
  namespace: demo-ns
spec:
  kubernetes:
    role: demo-ns-app
    serviceAccount: default
    tokenExpirationSeconds: 600
    audiences:
     - vault
  method: kubernetes
  mount: kubernetes
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: app-db-creds
  namespace: demo-ns
spec:
  destination:
    name: app-db-creds
    create: true
  mount: database
  path: creds/app-dbuser
  rolloutRestartTargets:
  - kind: Deployment
    name: app
  vaultAuthRef: app

Other useful info to include: kubectl describe deployment <app> and kubectl describe <vso-custom-resource> <app> output.

Expected behavior When VSO generates the JWT for the ServiceAccount it should (by default) include the Kubernetes APIServer audience for the cluster in the aud claim. The VaultDynamicSecret will then be created as expected.

Alternatively, this behavior could be configured by an additional parameter in the VaultAuth CRD to tell VSO that the client JWT is being used as the token reviewer JWT.

Environment

• Kubernetes version: 1.26.3
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): AKS
  • Other configuration options or runtime services (istio, etc.): istio 1.18.2
• vault-secrets-operator version: 0.2.0

Additional context

Workaround

For now, you can resolve this by manually adding the APIServer audience to the VaultAuth CustomResource. For example, for Azure Kubernetes Service this looks like:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: app
  namespace: demo-ns
spec:
  kubernetes:
    role: demo-ns-app
    serviceAccount: default
    tokenExpirationSeconds: 600
    audiences:
     - vault
     - "https://{REDACTED}.hcp.westeurope.azmk8s.io"
  method: kubernetes
  mount: kubernetes

If you are unsure of the APIServer audience for your cluster, use one of these examples to decode the JWT mounted in /run/secrets/kubernetes.io/serviceaccount/token for an existing pod and inspect the contents of the aud claim.

[benashz]

Hi @jtv8, thank you for the detailed report. We will definitely look into getting an ideal fix out to address the issue. If you could confirm that you have set the audience to vault on the k8s-auth role, that would be great!

Thanks,

Ben

[jtv8]

Hi @benashz - thanks for looking into this! Yes, I can confirm that the audience was set to vault on the role. During troubleshooting I tried both with and without - the resulting error message was the same, as I believe the issue occurs with Vault's authentication with the APIServer, and this would prevent the call to the TokenReview API which I assume is where Vault's token audience gets checked.

[telchak]

Hi @benashz @jtv8 ! Thank you very much for this feedback, you avoided me some headache ^^ For information, and for anyone having the same issue, I encontered the same error on a GKE cluster using this Kubernetes auth method.

For me, the audience string to add in VaultAuth resource was like that:

• https://container.googleapis.com/v1/projects/{PROJECT_ID}/locations/{ZONE/REGION}/clusters/{CLUSTER_NAME}

[anilpally]

@benashz @jtv8

Thanks for the issue, we have been facing the same issue, VSO - v0.5.1. I have tried the workaround, but no luck, please let me know if i am missing something here -

$cat vaultauth-default.yaml apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultAuth metadata: name: default-ns-vaultauth namespace: default spec: kubernetes: audiences: - "https://api.c31.be.ocp.dev.comp.com:6443" role: ocp_sample serviceAccount: default tokenExpirationSeconds: 600 method: kubernetes mount: kubernetes/c31-be-ocp-dev namespace: comp/si/global

=====

$ cat vaultstaticsecret-default.yaml

apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: name: vso-ex1 namespace: default spec: vaultAuthRef: default-ns-vaultauth mount: kv type: kv-v2 #namespace: c31-be-ocp-dev namespace: comp/si/global/ocp_sample path: ocp_sample refreshAfter: 60s destination: create: true name: vso-secret`

=== Vaultstaticsecret describe below ===

Spec: Destination: Create: true Name: vso-secret Overwrite: false Hmac Secret Data: true Mount: kv Namespace: comp/si/global/ocp_sample Path: ocp_sample Refresh After: 60s Type: kv-v2 Vault Auth Ref: default-ns-vaultauth Events: Type Reason Age From Message

---

Warning VaultClientConfigError 5m37s (x18 over 16m) VaultStaticSecret Failed to get Vault auth login: Error making API request.

Namespace: comp/si/global URL: PUT https://vault.ent.comp.com/v1/auth/kubernetes/c31-be-ocp-dev/login Code: 403. Errors:

• permission denied