Experimental Manual Rotation Of Vault Dynamic Secret Through VSO

[jaireddjawed]

Description

This is my project for Hack Week. I found this issue opened a few weeks ago, where the person who opened it would have liked to see this feature added and I wanted to see if it was possible. I managed to do it by creating a new annotation vault-secrets-operator/force-sync; once it's added to a dynamic secret, a sync for the secret is executed, allowing someone to manually rotate a dynamic secret.

Local Testing

1. Started Vault Server in Dev Mode in one terminal

vault server -dev -dev-root-token-id=root -log-level=debug

2. Enable Kubernetes Authentication in Vault for VSO

export K8S_HOST=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')


# Get the token from a service account you'll use
kubectl create serviceaccount vault-auth
kubectl create clusterrolebinding vault-auth-binding --clusterrole=system:auth-delegator --serviceaccount=default:vault-auth

# Create a service account token
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-secret
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
EOF

# Get the token
export SA_JWT_TOKEN=$(kubectl get secret vault-auth-secret -o jsonpath="{.data.token}" | base64 --decode)
export K8S_CACERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 --decode)

vault write auth/kubernetes/config \
  kubernetes_host="$K8S_HOST" \
  token_reviewer_jwt="$SA_JWT_TOKEN" \
  kubernetes_ca_cert="$K8S_CACERT" \
  issuer="https://kubernetes.default.svc.cluster.local"

# Create policy
vault policy write db-creds-policy - <<EOF
path "database/creds/my-db-role" {
  capabilities = ["read"]
}

path "database/roles/my-db-role" {
  capabilities = ["read"]
}

path "sys/leases/revoke" {
  capabilities = ["update"]
}

path "database/creds/*" {
  capabilities = ["read"]
}
EOF

# Create role
vault write auth/kubernetes/role/kubernetes-role \
  bound_service_account_names=vault-auth \
  bound_service_account_namespaces=default \
  policies=db-creds-policy \
  ttl=1h
  
# Confirm that kubernetes authentication was set up correctly
vault write auth/kubernetes/login role=kubernetes-role jwt=$SA_JWT_TOKEN'

3. Set up a dynamic secret in Vault by using the database secrets engine.

vault secrets enable database

vault write database/config/my_database \
    plugin_name=postgresql-database-plugin \
    allowed_roles="my-role" \
    connection_url="postgresql://{{username}}:{{password}}@127.0.0.1:5432/my_database" \
    username="admin" \
    password="admin-password"
   
vault write database/roles/my-role \
    db_name=my_database \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';" \
    default_ttl="20m" \
    max_ttl="30m"

4. Setup the VaultConnection, VaultAuth, and VaultDynamicSecret in Vault Secrets Operator

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vault-connection
  namespace: default
spec:
  address: http://VAULT_URL # Adjust this URL to your Vault server
  skipTLSVerify: true # For dev/test only - use proper TLS in production
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
  namespace: default
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: kubernetes-role # The Vault Kubernetes auth role you created
    serviceAccount: vault-auth # The service account to use
  vaultConnectionRef: vault-connection
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: db-creds
  namespace: default
spec:
  mount: database
  path: creds/my-role
  destination:
    name: postgres-creds
    create: true
  rolloutRestartTargets: []
  vaultAuthRef: vault-auth
  renewalPercent: 50
  revoke: true

5. View the initial base64-encoded password saved in VSO before the rotation

apiVersion: v1
data:
  _raw: eyJwYXNzd29yZCI6IkNEM2xFLTFJbjFucnN6cTdNNkdWIiwidXNlcm5hbWUiOiJ2LWt1YmVybmV0LW15LXJvbGUtMER1bWY2RXRKUEhkblk1UXd0TTItMTc0MTk0MDQ4NCJ9
  password: Q0QzbEUtMUluMW5yc3pxN002R1Y=
  username: di1rdWJlcm5ldC1teS1yb2xlLTBEdW1mNkV0SlBIZG5ZNVF3dE0yLTE3NDE5NDA0ODQ=
kind: Secret
metadata:
  creationTimestamp: "2025-03-14T08:21:24Z"
  labels:
    app.kubernetes.io/component: secret-sync
    app.kubernetes.io/managed-by: hashicorp-vso
    app.kubernetes.io/name: vault-secrets-operator
    secrets.hashicorp.com/vso-ownerRefUID: 5775f6fd-4cd0-4f77-b5e3-dba8befe1f88
  name: postgres-creds
  namespace: default
  ownerReferences:
  - apiVersion: secrets.hashicorp.com/v1beta1
    kind: VaultDynamicSecret
    name: db-creds
    uid: 5775f6fd-4cd0-4f77-b5e3-dba8befe1f88
  resourceVersion: "2297"
  uid: 0e99e731-8c08-412f-9305-e48d64ead271
type: Opaque

6. Added the annotation to trigger a manual rotation

kubectl annotate vaultdynamicsecrets.secrets.hashicorp.com db-creds vault-secrets-operator/force-sync="true" --overwrite

7. Confirm that the value for the password in db-creds changed

apiVersion: v1
data:
  _raw: eyJwYXNzd29yZCI6IlBmSGZGMnZsanNTUVQtN1JyUURoIiwidXNlcm5hbWUiOiJ2LWt1YmVybmV0LW15LXJvbGUtT1FRamNWYzVvVEpHcFZNcUkwRm4tMTc0MTk0MDcxMyJ9
  password: UGZIZkYydmxqc1NRVC03UnJRRGg=
  username: di1rdWJlcm5ldC1teS1yb2xlLU9RUWpjVmM1b1RKR3BWTXFJMEZuLTE3NDE5NDA3MTM=
kind: Secret
metadata:
  creationTimestamp: "2025-03-14T08:21:24Z"
  labels:
    app.kubernetes.io/component: secret-sync
    app.kubernetes.io/managed-by: hashicorp-vso
    app.kubernetes.io/name: vault-secrets-operator
    secrets.hashicorp.com/vso-ownerRefUID: 5775f6fd-4cd0-4f77-b5e3-dba8befe1f88
  name: postgres-creds
  namespace: default
  ownerReferences:
  - apiVersion: secrets.hashicorp.com/v1beta1
    kind: VaultDynamicSecret
    name: db-creds
    uid: 5775f6fd-4cd0-4f77-b5e3-dba8befe1f88
  resourceVersion: "2712"
  uid: 0e99e731-8c08-412f-9305-e48d64ead271
type: Opaque