vault icon indicating copy to clipboard operation
vault copied to clipboard
verified publisher icon hashicorp

The vault agent sidecar does not reuse from persistent cache the token fetched by the initcontainer agent.

Open ozozgun opened this issue 1 year ago • 0 comments

Hello,

I have a postgreSQL DB running on VMs that I want to access from a java app running on K8s. I want to use the dynamic database secrets generated by vault. The dynamic secret configuration works well. I can manually fetch the secrets from vault UI and connect to the DB. I am now having issue to implement this for my app. ⚠️ I cannot use the Agent Sidecar Injector from https://github.com/hashicorp/vault-k8s, my K8s private cluster does not allow any custom operator. I am using an initcontainer vault agent with templating for the output of secrets, a persistent cache to share the leases with the sidecar which will later renew the secrets when expired by restarting the app (template command).

Describe the bug

  1. I have a initcontainer auto-auth with K8s service token to vault
  2. initcontainer fetches the sercrets and putting in a file.
  3. The app reads the secrets and starts well.
  4. But the sidecar reauthenticate to vault, fetch again the secrets and restarts the application right away.

To Reproduce here is my agent-config.hcl:

pid_file = "/home/vault/.pid"

auto_auth {
  method "kubernetes" {
    mount_path = "auth/kubernetes"
    namespace = "myVault/OZ_VAULT"
    config = {
      role = "kube-role"
    }
  }
  
  sink "file" {
    config = {
      path = "/home/vault/.vault-token"
    }
  }
}

cache {
  use_auth_auth_token = "true"
  enforce_consistency = "always"
  persist "kubernetes" {
    path = "/vault/agent-cache"
  }
}

vault {
  address = "https://vault-test-local"
  ca_cert = "/etc/pki/unipass-server.pem"
}

template {
  destination = "/vault/secrets/application-vault.yaml"
  contents = <<EOT
    {{- with secret "database/creds/read_write_role" }}
MY_DB_USER: {{ .Data.username }}
MY_DB_PASSWORD: {{ .Data.password }}
  {{ end }}
  EOT
  command     = "/bin/sh -c \"kill -TERM $(pidof java) || true\""
}

I am using the vault image version 1.15.4 both for my vault server and for the agent containers. Running the containers (init and sidecar) with following command: vault agent -config=/vault/config/vault-agent-init-config.hcl -exit-after-auth (the sidecar does not have the -exit-after-auth flag) Kubernetes version: v1.24.6-mirantis-1

Here is the logs from my initcontainer:

2024-07-10T13:19:23.209+0200 [INFO]  agent.sink.file: creating file sink
2024-07-10T13:19:23.209+0200 [INFO]  agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2024-07-10T13:19:23.209+0200 [INFO]  agent.cache: configured persistent storage: path=/vault/agent-cache
2024-07-10T13:19:23.209+0200 [INFO]  agent.exec.server: starting exec server
2024-07-10T13:19:23.209+0200 [INFO]  agent.exec.server: no env templates or exec config, exiting
2024-07-10T13:19:23.209+0200 [INFO]  agent.auth.handler: starting auth handler
2024-07-10T13:19:23.210+0200 [INFO]  agent.sink.server: starting sink server
==> Note: Vault Agent version does not match Vault server version. Vault Agent version: 1.15.4, Vault server version: 1.15.4+ent.hsm
==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

           Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.15.4, built 2023-12-04T17:45:28Z
             Version Sha: 9b61934559ba31150860e618cf18e816cbddc630

2024-07-10T13:19:23.210+0200 [INFO]  agent.auth.handler: authenticating

And here is my sidecar when it starts:

==> Note: Vault Agent version does not match Vault server version. Vault Agent version: 1.15.4, Vault server version: 1.15.4+ent.hsm
==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

           Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.15.4, built 2023-12-04T17:45:28Z
             Version Sha: 9b61934559ba31150860e618cf18e816cbddc630

2024-07-10T13:19:25.672+0200 [INFO]  agent.sink.file: creating file sink
2024-07-10T13:19:25.672+0200 [INFO]  agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2024-07-10T13:19:25.673+0200 [INFO]  agent.cache: loaded memcache from persistent storage
2024-07-10T13:19:25.673+0200 [INFO]  agent.exec.server: starting exec server
2024-07-10T13:19:25.673+0200 [INFO]  agent.exec.server: no env templates or exec config, exiting
2024-07-10T13:19:25.673+0200 [INFO]  agent.sink.server: starting sink server
2024-07-10T13:19:25.673+0200 [INFO]  agent.template.server: starting template server
2024-07-10T13:19:25.673+0200 [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-07-10T13:19:25.673+0200 [INFO]  agent.auth.handler: starting auth handler
2024-07-10T13:19:25.674+0200 [INFO]  agent.auth.handler: authenticating
2024-07-10T13:19:25.674+0200 [INFO]  agent: (runner) creating watcher
2024-07-10T13:19:25.892+0200 [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2024-07-10T13:19:25.892+0200 [INFO]  agent.auth.handler: starting renewal process
2024-07-10T13:19:25.892+0200 [INFO]  agent.template.server: template server received new token
2024-07-10T13:19:25.892+0200 [INFO]  agent: (runner) stopping
2024-07-10T13:19:25.892+0200 [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-07-10T13:19:25.892+0200 [INFO]  agent.sink.file: token written: path=/home/vault/.vault-token
2024-07-10T13:19:25.892+0200 [INFO]  agent: (runner) creating watcher
2024-07-10T13:19:25.892+0200 [INFO]  agent: (runner) starting
2024-07-10T13:19:25.895+0200 [INFO]  agent.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/database/creds/read_write_role
2024-07-10T13:19:25.895+0200 [INFO]  agent.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/database/creds/read_write_role
2024-07-10T13:19:25.906+0200 [INFO]  agent.auth.handler: renewed auth token
2024-07-10T13:19:25.913+0200 [INFO]  agent.apiproxy: received request: method=GET path=/v1/database/creds/read_write_role
2024-07-10T13:19:25.913+0200 [INFO]  agent.apiproxy: forwarding request to Vault: method=GET path=/v1/database/creds/read_write_role
2024-07-10T13:19:26.072+0200 [INFO]  agent.apiproxy: received request: method=PUT path=/v1/sys/leases/renew
2024-07-10T13:19:26.072+0200 [INFO]  agent.apiproxy: forwarding request to Vault: method=PUT path=/v1/sys/leases/renew
2024-07-10T13:19:26.073+0200 [INFO]  agent: (runner) rendered "(dynamic)" => "/vault/secrets/application-vault.yaml"
2024-07-10T13:19:26.073+0200 [INFO]  agent: (runner) executing command "[\"/bin/sh -c \\\"kill -TERM $(pidof java) || true\\\"\"]" from "(dynamic)" => "/vault/secrets/application-vault.yaml"
2024-07-10T13:19:26.073+0200 [INFO]  agent: (child) spawning: sh -c /bin/sh -c "kill -TERM $(pidof java) || true"

Expected behavior I would the sidecar to reuse the leases from the cache created by the initcontainer and restarts the application only the currently used credential expires, instead of fetching new secrets and restarting the app right away.

Additional context Add any other context about the problem here.

ozozgun avatar Jul 10 '24 11:07 ozozgun