hashicorp
Cannot operate on existing KV entries that contains spaces in the key
Overview of the Issue
We've been running Consul for some time now. We have an automated script to place our ACME certs in the KV store for use with Traefik, and we opted for the key naming convention "acme - hostname". Since upgrading to v1.21.5, spaces are no longer allowed in key names. We recreated all the entries using an underscore instead, but now we cannot delete the existing entries containing spaces. We get the following error:
There was an error saving your key.
500: invalid key name, keys should respect the "^[a-zA-Z0-9,_./\\-?&=]+$" format.
This applies to both API calls and CLI usage.
Reproduction Steps
Steps to reproduce this issue, eg:
- Create a cluster using v1.21.4
- Create KV entries containing spaces in the key.
- Upgrade to v1.21.5
- Try to view, modify, or delete the key.
Consul info for both Client and Server
Server info
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = 3261d11a
version = 1.21.5
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = 10.60.30.11:8300
server = true
raft:
applied_index = 25396266
commit_index = 25396266
fsm_pending = 0
last_contact = 41.669233ms
last_log_index = 25396266
last_log_term = 26
last_snapshot_index = 25396214
last_snapshot_term = 26
latest_configuration = [{Suffrage:Voter ID:2e96dfcc-f51b-84a9-44e9-d05151cc07f7 Address:10.60.30.11:8300} {Suffrage:Voter ID:b808679a-a180-e5ca-3f8f-486c2d108cbc Address:10.60.30.4:8300} {Suffrage:Voter ID:58c895df-a7b4-76ca-fbaa-aa06925b9bc2 Address:10.60.30.12:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 26
runtime:
arch = amd64
cpu_count = 2
goroutines = 3976
max_procs = 2
os = linux
version = go1.25.0
Operating system and Environment details
ORACLE_BUGZILLA_PRODUCT="Oracle Linux 9" ORACLE_BUGZILLA_PRODUCT_VERSION=9.6 ORACLE_SUPPORT_PRODUCT="Oracle Linux" ORACLE_SUPPORT_PRODUCT_VERSION=9.6
You might also want to check that Traefik is still pulling the certs from Consul, as it's likely that it won't be able to get data from Consul after a restart, resulting in a high risk of an outage. Traefik has a healthcheck that checks if a key exists and considers Consul as down if it gets any errors, but unfortunately that key doesn't match the new regex so it always marks Consul as down.
Related Traefik issue: https://github.com/traefik/traefik/issues/12092 Other related Consul issue to this: https://github.com/hashicorp/consul/issues/22822
I mentioned this over on #22822, but you should be able to control whether the check runs or not via this agent config key: https://developer.hashicorp.com/consul/docs/reference/agent/configuration-file/general#disable_http_unprintable_char_filter
Thanks for mentioning this here as well! That did the trick and I was able to safely delete the old keys.
Not sure if anyone else would like to chime in on the PR improving this KV validation feature
https://github.com/hashicorp/consul/pull/22863
This stung me badly yesterday! I have a vault instance backed by consul, with auth paths at e.g. auth/Entra ID, auth/Google Cloud. Spent well over an hour trying to figure out why Vault wouldn't unseal...
Thanks to @dcarbone for posting the link to the disable_http_unprintable_char_filter flag, which allowed me to boot up Vault / Consul and then run e.g. vault auth move "auth/Entra ID" auth/entra_id
