vault
vault copied to clipboard
hashicorp
Multiple instances read key failed(/transit/keys/:name)
Describe the bug 1.Both vault services use the same mysql database,and vault Sealed is false. 2.when service A use client create aes256-gcm96 use default param(client.Logical().Write("transit/keys/vault_key", nil)) create key success and success get key info. {"request_id":"dd5ea9f5-01ff-60f1-600c-682dd2a36a9f","lease_id":"","lease_duration":0,"renewable":false,"data":{"allow_plaintext_backup":false,"auto_rotate_period":0,"deletion_allowed":false,"derived":false,"exportable":false,"imported_key":false,"keys":{"1":1656056877},"latest_version":1,"min_available_version":0,"min_decryption_version":1,"min_encryption_version":0,"name":"vault_key","supports_decryption":true,"supports_derivation":true,"supports_encryption":true,"supports_signing":false,"type":"aes256-gcm96"},"warnings":null}". 3.service A use this key encrypt(client.Logical().Write("transit/encrypt/vault_key", originalData)) data success. 4.After service A create the key, service B starts and get the key information(client.Logical().Read("transit/keys/vault_key"))within 2 seconds.the service B get information is null. 5.if service B get information failed,Continue to decrypt(client.Logical().Write("transit/decrypt/vault_key", encData)) encrypted data for service A,return error info "message authentication failed" and http code is 500.and continue get key information found "keys":{"1":1656056877}, change to "keys":{"1":1656056879},Other key information is the same as before. 6.restart the service A and get the key infomation((client.Logical().Read("transit/keys/vault_key"))),the key information also change to "keys":{"1":1656056879},Other key information is the same as before.
Environment:
- Vault Server Version (retrieve with
vault status):v.1.11.0 - Vault CLI Version (retrieve with
vault version):v1.0.4 - Server Operating System/Architecture: NAME="Alpine Linux" ID=alpine VERSION_ID=3.14.6 PRETTY_NAME="Alpine Linux v3.14" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://bugs.alpinelinux.org/"
Service A and service B are connected to the vault using the same configuration Vault server configuration file(s):
disable_mlock = true
ui = false
disable_cache = true
cluster_addr = "http://POD_IP:8201"
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "mysql" {
database = "vault918"
table = "vault"
username = "vault"
password = "xxxxx"
address = "10.10.10.240:3306"
ha_enabled = "false"
max_parallel = "128"
max_idle_connections = "50"
max_open_connections = "200"
max_connection_lifetime = "120"
}
telemetry {
prometheus_retention_time = "30s"
disable_hostname = true
}
# Example configuration for using auto-unseal, using Google Cloud skm. The
# Gskm keys must already exist, and the cluster must have a service account
# that is authorized to access GCP skm.
#seal "gcpcskm" {
# project = "vault-helm-dev-246514"
# region = "global"
# key_ring = "vault-helm-unseal-kr"
# crypto_key = "vault-helm-unseal-key"
Additional context
Add any other context about the problem here.
It sounds like you are running two entirely separate Vault servers, at the same time, pointing at the same database.
That's not a supported configuration, and explains the behaviour you are seeing.
Vault is a stateful application, and you cannot run multiple instances of it reading and writing to the same storage, unless those nodes are part of the same Vault HA cluster, in which case the Vault server processes co-operate to elect exactly one of them which will process requests at any one time.
@WeWebsterchen Max is correct - please let us know if changing your configuration to Vault HA, or not having two separate clusters solves the problem. Thanks!
@WeWebsterchen Max is correct - please let us know if changing your configuration to Vault HA, or not having two separate clusters solves the problem. Thanks!
when multiple instances The problem is sporadic，I understand that this has nothing to do with whether the vault has a state, because it does not write keys at the same time, it can be considered as a master-slave relationship.If set a single instance, there is no problem.
Just to be a bit more clear, two Vault servers pointing at the same database is likely to fail in all kinds of interesting ways. They need to be setup in HA mode so they're aware of each other and only one is the active node at any given time.
As per @sgmiller's comment, I'm going to go ahead and close this issue out. Please feel free to open a new one if you have problems with clusters in HA mode.