hashicorp
Vault agent silently DoSes Vault server when token policy does not include auth/token/renew-self
Summary
When the capability for a token to renew itself is missing from a renewable token's policies, vault agent silently DoS's the Vault server with failing requests to auth/token/renew-self which result in error: 1 error occurred:\n\t* permission denied\n\n with a 403 return code.
There are no errors reported from Vault Agent and there appears to be no backoff present as it is issuing hundreds of requests a second to Vault Server.
How to Reproduce
Pull the hashicorp/vault:1.14.1 Docker image - we will use this as both the server and the client.
docker pull hashicorp/vault:1.14.1
Start a container acting as Vault Server. For ease of inter-container networking, we define a custom network as the default bridge network provided by Docker does not have container-DNS for some reason.
docker network create vault
docker run --rm --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=hvault-root' -p 8200:8200 --network vault --name vault-server hashicorp/vault:1.14.1
Start an interactive Vault container to use as the client. Note that it is also on the custom vault network so that it can communicate with the server. Also configure it so that it can talk to Vault Server using the root token:
docker run --rm -it --network vault hashicorp/vault:1.14.1 ash
/ # export VAULT_ADDR=http://vault-server:8200
/ # export VAULT_TOKEN=hvault-root
/ # vault token lookup
Key Value
--- -----
accessor 5YfkHXssoq9bM2MnkajuboWt
creation_time 1691319976
...
policies [root]
renewable false
ttl 0s
type service
Next we need to alter the default policy to remove the ability for tokens to renew themselves. This can be done by using the Vault CLI for reading, editing, and writing the default policy but it can also be done via the web ui at http://localhost:8200. The policy should be edited to remove or comment out this capability:
# Allow tokens to renew themselves
path "auth/token/renew-self" {
capabilities = ["update"]
}
As Vault Agent does not log any errors, we need to enable audit logging in Vault Server so that we can see the deluge of requests arriving from Vault agent:
vault audit enable file file_path=stdout
We are now nearly ready to launch vault agent. We need to create a renewable service token with the default policy, put it in a file, create the Vault agent config file, and then start Vault Agent:
# Create vault token and write to file
vault token create -ttl 600 -renewable -policy default -field token > /tmp/vault-token
# Write out vault agent config file
cat >/tmp/vault-agent.conf <<EOL
vault {
address = "https://vault-fqdn:8200"
retry {
num_retries = 5
}
}
auto_auth {
method {
type = "token_file"
config = {
token_file_path = "/tmp/vault-token"
}
}
sink "file" {
config = {
path = "/tmp/new-vault-token"
}
}
}
log_level = "trace"
EOL
# Start vault agent
vault agent -config /tmp/vault-agent.conf
At this point, the audit log in the stdout of the Vault Server instance will become very active with a continuous stream of requests for the token to renew itself but the Vault agent log will remain quiet even though trace logging is enabled. Restoring the capability to update auth/token/renew-self to the default policy stops the deluge of requests from vault agent and it appears to operate as normal afterwards.
Hey there! Was chatting about this internally. I recognize this is an old issue but I thought I'd update it regardless.
One notable is that you mark the token as renewable:
vault token create -ttl 600 -renewable -policy default -field token > /tmp/vault-token
In particular, -renewable marks the token as renewable. For a renewable token, we try to do renew-self to renew the token, as we have no other mechanism available to us to renew the token. I agree that we should be more resilient in this case, but I wanted to indicate why this happens and a workaround in the meantime. Without renew-self, we'll be unable to renew the token, but also, we shouldn't silently DoS the server :)
We've got a bug raised internally for this and we're hoping to address it soon.
Wanted to give a heads up that I've reproduced this issue and am actively working on it. Thanks for the detailed report -- it really helps making it easier to find and debug.