hashicorp
[BUG] Vault Proxy static secrets cache does not consider secrets version
Describe the bug
When using the Vault Proxy with cache_static_secrets: true (ref. docs) we are observing inconsistent responses that we believe to have traced back to the cache key of the Vault Proxy's static secrets caching.
This is a critical issue & blocker for us trying to adopt the Vault Proxy in Vault Enterprise
The issue is that the cache key does not consider the version parameter when requesting secrets from a versioned secrets engine. This results in clients receiving outdated and sometimes inconsistent responses from the Vault Proxy, regardless of whether they specify the desired version (with ?version=N in the URL or -version N in the CLI) or not.
This function considers only the request Path and the X-Vault-Namespace header when it is set, however it needs to also consider query parameters such as version in order to work for versioned storage engines (ref. https://developer.hashicorp.com/vault/tutorials/secrets-management/versioned-kv).
// getStaticSecretPathFromRequest gets the canonical path for a
// request, taking into account intricacies relating to /v1/ and namespaces
// in the header.
// Returns a path like foo/bar or ns1/foo/bar.
// We opt for this form as namespace.Canonicalize returns a namespace in the
// form of "ns1/", so we keep consistent with path canonicalization.
func getStaticSecretPathFromRequest(req *SendRequest) string {
path := req.Request.URL.Path
// Static secrets always have /v1 as a prefix. This enables us to
// enable a pass-through and never attempt to cache or view-from-cache
// any request without the /v1 prefix.
if !strings.HasPrefix(path, "/v1") {
return ""
}
var namespace string
if header := req.Request.Header; header != nil {
namespace = header.Get(api.NamespaceHeaderName)
}
return canonicalizeStaticSecretPath(path, namespace)
}
code ref. https://github.com/hashicorp/vault/blob/main/command/agentproxyshared/cache/lease_cache.go#L888-L907
The cache key for dynamic secrets does not seem to have this issue (code ref. https://github.com/hashicorp/vault/blob/main/command/agentproxyshared/cache/lease_cache.go#L837-L862)
To Reproduce
Steps to reproduce the behavior:
- Run
vault kv get -namespace {namespace} -version 1 {kv_name}/{secret_path}- returned version 1 (expected, as we requested version 1)
2024-08-21T12:07:55.654Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:07:55.654Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:07:55.654Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:07:55.654Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:07:55.654Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{remote_vault}:8200/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:07:55.850Z [DEBUG] proxy.cache.leasecache: pass-through response; secret not renewable: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:07:55.851Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:07:55.851Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:07:55.851Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:07:55.851Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:07:55.851Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{remote_vault}:8200/v1/{kv_name}/data/{secret_path}?version=1
2024-08-21T12:07:56.040Z [TRACE] proxy.cache.leasecache: attempting to cache static secret with following request path: request path={namespace}/{kv_name}/data/{secret_path}
2024-08-21T12:07:56.040Z [DEBUG] proxy.cache.leasecache: storing static secret response into the cache: method=GET path={namespace}/{kv_name}/data/{secret_path} id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7
- Run
vault kv get -namespace {namespace} -version 3 {kv_name}/{secret_path}- returned version 1 (wrong, as we requested version 3)
2024-08-21T12:08:00.973Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}/{secret_path}
2024-08-21T12:08:00.973Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}/{secret_path}
2024-08-21T12:08:00.973Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}/{secret_path}
2024-08-21T12:08:00.973Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}/{secret_path}
2024-08-21T12:08:00.973Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{remote_vault}:8200/v1/sys/internal/ui/mounts/{kv_name}/{secret_path}
- Run
vault kv get -namespace {namespace} {kv_name}/{secret_path}- Returned version 1 (wrong, as we didn't specify a version and expect the latest which is version 3)
2024-08-21T12:08:01.161Z [DEBUG] proxy.cache.leasecache: pass-through response; secret not renewable: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}/{secret_path}
2024-08-21T12:08:01.162Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:08:01.162Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:08:01.162Z [DEBUG] proxy.cache.leasecache: returning cached static secret response: id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7 path={namespace}/{kv_name}/data/{secret_path}
2024-08-21T12:08:05.571Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:08:05.571Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:08:05.571Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:08:05.571Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:08:05.571Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{remote_vault}:8200/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:08:05.754Z [DEBUG] proxy.cache.leasecache: pass-through response; secret not renewable: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:08:05.755Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:08:05.755Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:08:05.755Z [DEBUG] proxy.cache.leasecache: returning cached static secret response: id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7 path={namespace}/{kv_name}/data/{secret_path}
- Run
vault kv get -namespace {namespace} {kv_name}/{secret_path}- Returned version 3 (expected, latest - inconsistent response to the same request)
2024-08-21T12:12:09.439Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:09.439Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:09.439Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:09.439Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:09.439Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{remote_vault}:8200/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:09.621Z [DEBUG] proxy.cache.leasecache: pass-through response; secret not renewable: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:09.623Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:12:09.623Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:12:09.623Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:12:09.623Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:12:09.623Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{remote_vault}:8200/v1/{kv_name}/data/{secret_path}
2024-08-21T12:12:09.817Z [TRACE] proxy.cache.leasecache: attempting to cache static secret with following request path: request path={namespace}/{kv_name}/data/{secret_path}
2024-08-21T12:12:09.818Z [DEBUG] proxy.cache.leasecache: storing static secret response into the cache: method=GET path={namespace}/{kv_name}/data/{secret_path} id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7
- Run
vault kv get -namespace {namespace} {kv_name}/{secret_path}- Returned version 1 (wrong, expected the latest - inconsistent response)
2024-08-21T12:12:12.220Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:12.220Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:12.220Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:12.220Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:12.220Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{remote_vault}:8200/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:12.553Z [DEBUG] proxy.cache.leasecache: pass-through response; secret not renewable: method=GET path=/v1/sys/internal/ui/mounts/{kv_name}
2024-08-21T12:12:12.554Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:12:12.554Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/{kv_name}/data/{secret_path}
2024-08-21T12:12:12.555Z [DEBUG] proxy.cache.leasecache: returning cached static secret response: id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7 path={namespace}/{kv_name}/data/{secret_path}
Expected behavior
- When version is specified, for Vault Proxy cache to return the correct requested version of the secret.
- When version is not specified, for Vault Proxy cache to return the latest version of the requested secret.
Environment:
- Vault Server Version:
1.17.3 - Vault Proxy Version:
1.17.3 - Vault CLI Version:
1.17.3 - Server Operating System/Architecture: Linux / amd64
- Proxy Operating System/Architecture: macOS / arm64
Vault server configuration file(s):
# ref. https://developer.hashicorp.com/vault/docs/agent-and-proxy/autoauth
auto_auth {
method "approle" {
mount_path = "auth/approle"
config = {
role_id_file_path = "/etc/vault/role_id" # secrets expected to be moounted as volumes
secret_id_file_path = "/etc/vault/secret_id" # secrets expected to be moounted as volumes
remove_secret_id_file_after_reading = false
}
}
}
# ref. https://developer.hashicorp.com/vault/docs/agent-and-proxy/proxy/apiproxy
api_proxy {
use_auto_auth_token = "force"
enforce_consistency = "always"
when_inconsistent = "retry"
}
# ref. https://developer.hashicorp.com/vault/docs/agent-and-proxy/proxy/caching
cache {
# ref. https://developer.hashicorp.com/vault/docs/agent-and-proxy/proxy/caching/static-secret-caching
cache_static_secrets = true
static_secret_token_capability_refresh_interval = "1m"
static_secret_token_capability_refresh_behavior = "optimistic"
}
# ref. https://developer.hashicorp.com/vault/docs/configuration/listener/tcp
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true # http local connections
}
# ref. https://developer.hashicorp.com/vault/docs/agent-and-proxy/proxy#vault-stanza
vault {
# vault address is configured by means of the VAULT_ADDR environment variable
# vault namespace is configured by means of the VAULT_NAMESPACE environment variable
tls_skip_verify = true
}
Running the Vault Proxy (version 1.17.3) with Docker:
docker run --rm -it \
-p 8200:8200 \
-v "$(pwd)/vault-agent/secrets/role_id:/etc/vault/role_id" \
-v "$(pwd)/vault-agent/secrets/secret_id:/etc/vault/secret_id" \
-v "$(pwd)/vault-agent/config/vault-proxy-config.hcl:/etc/vault/vault-proxy-config.hcl" \
-e VAULT_ADDR=https://{remote_vault}:8200 \
-e VAULT_NAMESPACE={namespace} \
--hostname $(hostname) \
hashicorp/vault:1.17.3 \
proxy -config /etc/vault/vault-proxy-config.hcl -log-level trace
Additionally it seems like the Vault Proxy documentation on request uniqueness describes the implementation of the cache key as:
In order to detect repeat requests and return cached responses, Proxy needs to have a way to uniquely identify the requests. This computation as it stands today takes a simplistic approach (may change in future) of serializing and hashing the HTTP request along with all the headers and the request body. This hash value is then used as an index into the cache to check if the response is readily available.
This is true for the implementation of dynamic secrets caching, however not for static secrets caching. Is there a different section that describes the latter?
Hey! Thanks for reporting this bug. I agree that this is a bug and something we should fix. I can't promise timelines but I'll raise something internally and try and get this prioritized.
In short, it looks like what's happening here is as you described -- if I specify 'version 1' for a multi-version secret then Proxy ignores the version and caches that, then returns version 1 no matter which version I specify later.
This is true for the implementation of dynamic secrets caching, however not for static secrets caching. Is there a different section that describes the latter?
For static secrets, we do not cache based on a key formed by the request body/headers like we do dynamic secrets. There are multiple cache keys (we have one for the secret and one for the token), but ultimately each static secret should be cached a single time, and unlike dynamic secrets I feel like this is more of a superfluous implementation detail (not that it's a secret, and more that the average user shouldn't need to know). The high functionality/implementation details are documented here, though: https://developer.hashicorp.com/vault/docs/agent-and-proxy/proxy/caching/static-secret-caching#functionality
Also, thanks for the great and detailed bug report. It really helps us internally both understand the issue and reproduce it. It's greatly appreciated :)
Thanks @VioletHynes for your quick triaging and confirmation of this bug!
For static secrets, we do not cache based on a key formed by the request body/headers like we do dynamic secrets. There are multiple cache keys (we have one for the secret and one for the token), but ultimately each static secret should be cached a single time, and unlike dynamic secrets I feel like this is more of a superfluous implementation detail (not that it's a secret, and more that the average user shouldn't need to know)
I feel like there should be a cache for every version of a given secret, as different clients may be interested in different versions of the secret. If one client happens to request version 1, that secret version should not be retrieved from the cache when another client requests the secret with no version specified (expecting the latest). IMO version 1 should also be cached, ideally with an optionally different caching configuration in order to be able to evict such "non-current" more aggressively than "latest / current".
I really appreciate your help with this!
Hey @carlzogh ! Wanted to check back in to say that I've been working on this, and a fix for this should be coming soon. Once it's merged, I'll close this and update with which versions you'll be able to find the fix in.
Thank you so much @VioletHynes - looking forward to that!
Are you able to specify whether we will need both the remote Vault Server + Vault Proxy to consume the new version for the fix to work, or would a Proxy / Agent version update alone work?
Hi there!
Happy to report that the fix for this has been merged and backported to older releases.
As a result of this change, versions will be correctly handled by Proxy's static secret caching. In particular:
- Versions will be cached alongside the latest version of a secret
- When a 'latest' secret is cached, the corresponding cached version is updated with it (e.g. if my latest version is 3, and I GET
/secrets/foo/my-secret, then the GET to/secrets/foo/my-secret?version=3will be a cache hit) - When a Proxy receives an event indicating a new version of a secret, the corresponding cached version is updated with it (e.g. if my latest version is 2, and the secret gets updated to secret 3, then a request for either the latest version or the cached version 3 will result in a cache hits)
- Proxy will also handle
delete,undelete, anddestroyrequests for versions, purging from the cache where appropriate.
This fix should be present in the following releases: 1.18.0 (CE and Ent) 1.17.5 (CE and Ent) 1.16.9 (Ent only release)
This fix is Proxy specific, and only Proxy will need to be upgraded for this behaviour to be fixed (i.e. server does not need an upgrade).
Thanks for the great bug report!
Hey @VioletHynes - thank you again for addressing the reported issue!
This patch has however changed the behavior of the Proxy in a way we didn't expect and likely puts a spanner in the works for our current design strategy.
What we were designing for
Our intention was to use the Vault Proxy as an interface into a remote self-hosted Vault Server, in an environment where applications need to able to rely on the Vault Proxy to operate and return cached secret responses when Vault Server is unreachable from Vault Proxy.
We intend on requesting a "known" set of secrets from Vault Proxy on a regular basis (eg. every few minutes) to ensure its cache is up-to-date and that it is able to serve these secrets to applications that require them. It is crucial in our design that:
- Vault Proxy does not fail client requests when Vault Server is unreachable and the secret version explicitly requested is available in the Vault Proxy cache. There is no need to reach out to Vault Server when the artifact is available locally.
- Vault Proxy does not fail client requests when Vault Server is unreachable and the latest version of the secret is requested. It would be ideal to be able to configure a retry strategy in such cases, to avoid client requests timing out while the Proxy is (unsuccessfully) retrying requests for "latest" from the Server.
- Vault Proxy is able to recover its persisted cache completely offline, start up and successfully serve secrets that are cached, without requiring a synchronous connection to Vault Server.
Before these changes on version 1.17.3, all these 3 requirements were satisfied and our only hiccup was the bug reported in this issue where Vault Proxy would incorrectly cache and return the wrong versions of secrets.
Behavior observed in version 1.17.5
After upgrading our Vault Proxy to 1.17.5, we are now seeing a synchronous request to get the "latest" version of a requested secret from Proxy to Server. This request is always performed, regardless of whether we already have the "latest" version, or if we already have the explicit version of the requested secret.
Request to get secret version 3 with "fresh" / empty cache:
$ vault kv get -namespace {namespace} -mount {kvv2_name} -version 3 {secret_path}
... [expected secret data]
Vault Proxy makes 2 requests to Vault Server (first to get "latest" version of the secret, then to get the specific version 3):
2024-09-03T07:42:15.662Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:15.663Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:15.663Z [TRACE] proxy.cache.leasecache: checking cache for dynamic secret request: id=ef89d0b9b08f3df7a46368c002ac84279134f4b8862d5e37afcf2dcac441b9b8
2024-09-03T07:42:15.663Z [TRACE] proxy.cache.leasecache: checking cache for static secret request: id=cf5a13814aa0b919e4bf4e5213ac6570675ce35421836b9721d7ee3a5815314b
2024-09-03T07:42:15.663Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:15.664Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:15.664Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:15.850Z [DEBUG] proxy.cache.leasecache: pass-through response; secret not renewable: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:15.852Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/{kvv2_name}/data/{secret_path}
2024-09-03T07:42:15.852Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/{kvv2_name}/data/{secret_path}
2024-09-03T07:42:15.853Z [TRACE] proxy.cache.leasecache: checking cache for dynamic secret request: id=bd33138218fc66fb4341d3408a5d47314dfd0ff35e3e2ff8086b16327dc35131
2024-09-03T07:42:15.853Z [TRACE] proxy.cache.leasecache: checking cache for static secret request: id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7
2024-09-03T07:42:15.853Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/{kvv2_name}/data/{secret_path}
2024-09-03T07:42:15.853Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/{kvv2_name}/data/{secret_path}
2024-09-03T07:42:15.853Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{vault_server}:8200/v1/{kvv2_name}/data/{secret_path}?version=3
2024-09-03T07:42:16.035Z [TRACE] proxy.cache.leasecache: attempting to cache static secret with following request path: request path={namespace}/{kvv2_name}/data/{secret_path} version=3
2024-09-03T07:42:16.036Z [DEBUG] proxy.cache.leasecache: storing static secret response into the cache: path={namespace}/{kvv2_name}/data/{secret_path} id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7
2024-09-03T07:42:16.038Z [TRACE] proxy.cache.leasecache: set entry in persistent storage: type=static-secret path={namespace}/{kvv2_name}/data/{secret_path} id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7
2024-09-03T07:42:16.039Z [TRACE] proxy.cache.leasecache: set entry in persistent storage: type=token-capabilities id=3b95578456a7eae66a1ddaee54ad4e8bcb63ff77d692a3ea04f9dc5739c91836
Request to get secret version 3 with the resulting cache containing version 3:
$ vault kv get -namespace {namespace} -mount {kvv2_name} -version 3 {secret_path}
... [expected secret data]
Vault Proxy makes 1 request to Vault Server to get "latest", then returns the cached version 3:
2024-09-03T07:42:58.898Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:58.898Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:58.898Z [TRACE] proxy.cache.leasecache: checking cache for dynamic secret request: id=ef89d0b9b08f3df7a46368c002ac84279134f4b8862d5e37afcf2dcac441b9b8
2024-09-03T07:42:58.898Z [TRACE] proxy.cache.leasecache: checking cache for static secret request: id=cf5a13814aa0b919e4bf4e5213ac6570675ce35421836b9721d7ee3a5815314b
2024-09-03T07:42:58.898Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:58.899Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:58.899Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:59.086Z [DEBUG] proxy.cache.leasecache: pass-through response; secret not renewable: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:42:59.088Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/{kvv2_name}/data/{secret_path}
2024-09-03T07:42:59.088Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/{kvv2_name}/data/{secret_path}
2024-09-03T07:42:59.088Z [TRACE] proxy.cache.leasecache: checking cache for dynamic secret request: id=bd33138218fc66fb4341d3408a5d47314dfd0ff35e3e2ff8086b16327dc35131
2024-09-03T07:42:59.088Z [TRACE] proxy.cache.leasecache: checking cache for static secret request: id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7
2024-09-03T07:42:59.088Z [DEBUG] proxy.cache.leasecache: returning cached static secret response: id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7 path={namespace}/{kvv2_name}/data/{secret_path}
Request to get secret version 3 offline:
For this scenario we emulate the Vault Server being unreachable to the Vault Proxy by temporarily disabling the network connection between them. As I am running the Vault Proxy locally on my host where I am also running the client command applications, I emulated this by simply turning off my Wifi connection before requesting the secret.
$ vault kv get -namespace {namespace} -mount {kvv2_name} -version 3 {secret_path}
context deadline exceeded
Vault Proxy attempts to request the "latest" from Vault Server however due to the lack of connectivity, this request continuously fails and retries. In the meantime, the client application is waiting synchronously on the Vault Proxy to complete the request to Vault Server (which it will not under these conditions), until the client times out before the Proxy does.
2024-09-03T07:43:33.645Z [INFO] proxy.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:43:33.646Z [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:43:33.646Z [TRACE] proxy.cache.leasecache: checking cache for dynamic secret request: id=ef89d0b9b08f3df7a46368c002ac84279134f4b8862d5e37afcf2dcac441b9b8
2024-09-03T07:43:33.646Z [TRACE] proxy.cache.leasecache: checking cache for static secret request: id=cf5a13814aa0b919e4bf4e5213ac6570675ce35421836b9721d7ee3a5815314b
2024-09-03T07:43:33.646Z [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:43:33.647Z [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:43:33.647Z [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:43:51.694Z [ERROR] proxy.apiproxy.client: request failed: error="Get \"https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}\": unexpected EOF" method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:43:51.695Z [DEBUG] proxy.apiproxy.client: retrying request: request="GET https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}" timeout=1.082765468s remaining=12
2024-09-03T07:43:52.786Z [ERROR] proxy.apiproxy.client: request failed: error="Get \"https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}\": dial tcp: lookup {vault_server} on {local_ip}:53: no such host" method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:43:52.786Z [DEBUG] proxy.apiproxy.client: retrying request: request="GET https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}" timeout=2.510020122s remaining=11
2024-09-03T07:43:55.305Z [ERROR] proxy.apiproxy.client: request failed: error="Get \"https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}\": dial tcp: lookup {vault_server} on {local_ip}:53: no such host" method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:43:55.305Z [DEBUG] proxy.apiproxy.client: retrying request: request="GET https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}" timeout=3.208266615s remaining=10
2024-09-03T07:43:58.520Z [ERROR] proxy.apiproxy.client: request failed: error="Get \"https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}\": dial tcp: lookup {vault_server} on {local_ip}:53: no such host" method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:43:58.520Z [DEBUG] proxy.apiproxy.client: retrying request: request="GET https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}" timeout=5.88085964s remaining=9
2024-09-03T07:44:04.410Z [ERROR] proxy.apiproxy.client: request failed: error="Get \"https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}\": dial tcp: lookup {vault_server} on {local_ip}:53: no such host" method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:44:04.410Z [DEBUG] proxy.apiproxy.client: retrying request: request="GET https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}" timeout=5.57765105s remaining=8
2024-09-03T07:44:09.996Z [ERROR] proxy.apiproxy.client: request failed: error="Get \"https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}\": dial tcp: lookup {vault_server} on {local_ip}:53: no such host" method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:44:09.996Z [DEBUG] proxy.apiproxy.client: retrying request: request="GET https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}" timeout=8.908386396s remaining=7
2024-09-03T07:44:18.911Z [ERROR] proxy.apiproxy.client: request failed: error="Get \"https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}\": dial tcp: lookup {vault_server} on {local_ip}:53: no such host" method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:44:18.911Z [DEBUG] proxy.apiproxy.client: retrying request: request="GET https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}" timeout=8.716394358s remaining=6
2024-09-03T07:44:27.635Z [ERROR] proxy.apiproxy.client: request failed: error="Get \"https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}\": dial tcp: lookup {vault_server} on {local_ip}:53: no such host" method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T07:44:27.635Z [DEBUG] proxy.apiproxy.client: retrying request: request="GET https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}" timeout=9.394666816s remaining=5
### at this point the client application times out and the Proxy drops the request and stops retrying.
Expected behavior
While we have not found Vault documentation stating that the Proxy's persisted cache will work offline, we expected this to be an implicit property of the system as it would have all the information it needs to produce valid responses to cached requests.
As a result of this change, versions will be correctly handled by Proxy's static secret caching. In particular:
- Versions will be cached alongside the latest version of a secret
- When a 'latest' secret is cached, the corresponding cached version is updated with it (e.g. if my latest version is 3, and I GET /secrets/foo/my-secret, then the GET to /secrets/foo/my-secret?version=3 will be a cache hit)
- When a Proxy receives an event indicating a new version of a secret, the corresponding cached version is updated with it (e.g. if my latest version is 2, and the secret gets updated to secret 3, then a request for either the latest version or the cached version 3 will result in a cache hits)
- Proxy will also handle delete, undelete, and destroy requests for versions, purging from the cache where appropriate.
This description of the modeled behavior after changes in this Issue do not necessarily imply the change in behavior we are observing.
Could you please advise and let us know if this something you could help with (eg. through Proxy configuration or a subsequent patch)?
Hey there! I agree that this shouldn't be happening. Apologies for missing this. Curiously, I did write a test with (what I believed to be) this specific use case, so I must have missed something about it. You're right in saying that this is a supported pattern.
Going to dig into this and try to fully understand the issue. To confirm: you're only observing this when attempting to retrieve specific versions of secrets?
Thanks again @VioletHynes - we are seeing this for requests both with and without a specific version - eg:
$ vault kv get -namespace {namespace} -mount {kvv2_name} {secret_path}
... [expected "latest" secret data]
Proxy logs:
2024-09-03T14:00:19.845+0100 [INFO] proxy.apiproxy: received request: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path} query=""
2024-09-03T14:00:19.845+0100 [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path} query=""
2024-09-03T14:00:19.845+0100 [TRACE] proxy.cache.leasecache: checking cache for dynamic secret request: id=23818249cc7a247646f8cc90e1ddcbb458b9bcc7c3212b56a9607c25e295f500
2024-09-03T14:00:19.845+0100 [TRACE] proxy.cache.leasecache: checking cache for static secret request: id=cf5a13814aa0b919e4bf4e5213ac6570675ce35421836b9721d7ee3a5815314b
2024-09-03T14:00:19.845+0100 [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path} query=""
2024-09-03T14:00:19.845+0100 [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path} query=""
2024-09-03T14:00:19.845+0100 [DEBUG] proxy.apiproxy.client: performing request: method=GET url=https://{vault_server}:8200/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path}
2024-09-03T14:00:20.039+0100 [DEBUG] proxy.cache.leasecache: pass-through response; secret not renewable: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path} query=""
2024-09-03T14:00:20.040+0100 [INFO] proxy.apiproxy: received request: method=GET path=/v1/{kvv2_name}/data/{secret_path} query=""
2024-09-03T14:00:20.040+0100 [DEBUG] proxy.apiproxy: using auto auth token: method=GET path=/v1/{kvv2_name}/data/{secret_path} query=""
2024-09-03T14:00:20.040+0100 [TRACE] proxy.cache.leasecache: checking cache for dynamic secret request: id=a445712ccfaa8bc33d477b38e29cd5abca57023318d610f7b524455751dda06b
2024-09-03T14:00:20.040+0100 [TRACE] proxy.cache.leasecache: checking cache for static secret request: id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7
2024-09-03T14:00:20.040+0100 [DEBUG] proxy.cache.leasecache: returning cached static secret response: id=b0b444679a0300f2cf23b576d35637fae0b93dfda757e1de9c2874eb4a50a7f7 path={namespace}/{kvv2_name}/data/{secret_path}
(note I've tested this with a local "vault" build where logs also include the "query" parameters for all requests, as the current output was sometimes misleading, in that versions are query string params that are not part of Request.URL.Path but of Request.URL.RawQuery or Request.URL.Query())
Would you be re-opening and using this same issue or would you like me to create a new one?
As long as it's cool with you let's stick here -- less tabs for me to track! I'll reopen it for now.
Curiously, I'm not noticing this behaviour at all. Are you certain that the requests you're making are for cached secrets?
In your logs above, the two lines here indicate that it's not cached:
2024-09-03T14:00:19.845+0100 [TRACE] proxy.cache.leasecache: checking cache for static secret request: id=cf5a13814aa0b919e4bf4e5213ac6570675ce35421836b9721d7ee3a5815314b
2024-09-03T14:00:19.845+0100 [DEBUG] proxy.cache.leasecache: forwarding request from cache: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path} query=""
2024-09-03T14:00:19.845+0100 [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path} query=""
If the secret is cached, these logs will look something like this (this request was served by the cache while Vault was sealed):
2024-09-03T09:02:11.544-0400 [TRACE] proxy.cache.leasecache: checking cache for static secret request: id=964ae42927a0e5f76fa3920205774ee234072f3111a9d3716a14fddbabe22b58
2024-09-03T09:02:11.544-0400 [DEBUG] proxy.cache.leasecache: returning cached static secret response: id=964ae42927a0e5f76fa3920205774ee234072f3111a9d3716a14fddbabe22b58 path=secret-v2/data/my-secret
Indicate that this is a cache miss. Proxy cannot serve a secret it doesn't have cached when Vault is offline.
I just attempted again with additional tests and I can't seem to reproduce this.
Oh! I see the issue:
2024-09-03T14:00:19.845+0100 [INFO] proxy.apiproxy: forwarding request to Vault: method=GET path=/v1/sys/internal/ui/mounts/{kvv2_name}/{secret_path} query=""
It's the preflight /sys/internal/ui/mounts request that the CLI is doing. This is an issue in previous versions and isn't something we can really 'solve'. To determine which mount the KV secret is on (KVv1 or KVv2), the CLI will do a request to that endpoint to check. This isn't a request we can cache.
To avoid this, you could use vault read, but you shouldn't be seeing this behaviour for any request to the secret.
I talked about this in a little more detail here: https://github.com/hashicorp/vault/issues/19879#issuecomment-1875813465
I'm going to re-close this issue. Please keep me updated if this does solve your issue but from what I've seen (and everything I've tested) I'm pretty sure it's specific to vault kv's preflight check and not the cache's contents. If this isn't your issue and we find a different issue/bug, I'm happy to re-open again.
That makes total sense, thanks! So it's a CLI-driven request, I can validate that I don't see this when requesting with curl:
curl -k -X GET http://127.0.0.1:8200/v1/{kvv2_name}/data/{secret_path} --header "X-Vault-Namespace: {namespace}"
Really appreciate the help with this!
Yeah, the 'preflight request' to /sys/internal/ui/mounts specific to vault kv (and will be true when interacting with any versions of Vault Proxy). Specifically what this additional request is doing is figuring out whether or not it's KVv1 or KVv2, and if it's KVv2 it'll add the /data/ part of the URL for you in some cases. In most situations this is a good thing to do, and the extra request is fine since generally the CLI isn't part of production workflows (direct API access is).
If you want to use the CLI to test it out, you can use vault read to do the request instead of vault kv, since that will avoid the preflight request.