vault

vault copied to clipboard

Describe the bug Setup : multi DC vault setup with consul as the storage backend and the consul data is being replicated using consul-replicate

Bug : vault caches all the responses (both correct and nil/error value) from the storage backend. If we try to read the value in the secondary DC before the replication, that data becomes unavailable even after successful replication

To Reproduce

Read a non-existent data in the secondary DC 1.1. Run vault read transit/keys/non-existent-key-1

Create two keys (one with same name as in step 1.1) in the primary DC 2.1. Run vault write -f transit/keys/non-existent-key-1 2.2. Run vault write -f transit/keys/non-existent-key-2

Read keys in the secondary cluster 3.1. Run vault read transit/keys/non-existent-key-1 3.2. Run vault read transit/keys/non-existent-key-2

Force leader election in the secondary cluster. Run the following on the active node. 4.1. Run vault operator step-down

Read keys in the secondary cluster 5.1. Run vault read transit/keys/non-existent-key-1 5.2. Run vault read transit/keys/non-existent-key-2

Expected behavior Step 3.1 should give error - “Error : No value found at transit/keys/non-existent-key-1”

While steps 3.2, 5.1 and 5.2 should not give errors. Proving that there is no issue with the replication.

Similar behavior is observed in other backends as well such as kv and database.

Environment:

• Vault Server Version (retrieve with vault status):

# ./vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.13.0-dev1
Build Date      n/a
Storage Type    consul
Cluster Name    vault-cluster-67dc2e08
Cluster ID      7c42e0a1-8be9-59d5-2ec5-2032a2b9be6a
HA Enabled      true
HA Cluster      https://10.51.114.216:8201
HA Mode         active
Active Since    2022-12-06T10:01:12.402497608Z

• Vault CLI Version (retrieve with vault version): Vault v1.13.0-dev1

• Server Operating System/Architecture:

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Vault server configuration file(s):

# primary DC
{
    "ui": true,
  "telemetry": {
    "statsd_address": "127.0.0.1:8125"
  },
  "storage": {
    "consul": {
      "address": "127.0.0.1:8500",
      "path": "vault",
      "obfuscate_paths": 1,
      "token": "e98xxxxx-8xxx-8xx5-xxx6-d2xxxa5f9xxx"
    }
  },
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
 },
  "plugin_directory": "/var/lib/fk-sec-vault/plugins"
}

# secondary DC
{
    "disable_cache": false,
    "ui": true,
  "telemetry": {
    "statsd_address": "127.0.0.1:8125"
  },
  "storage": {
    "consul": {
      "address": "127.0.0.1:8500",
      "path": "vault",
      "obfuscate_paths": 1,
      "token": "e98xxxxx-8xxx-8xx5-xxx6-d2xxxa5f9xxx",
	"consistency_mode": "strong",
	"cache_size": 1310720
    }
  },
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
 },
  "plugin_directory": "/var/lib/fk-sec-vault/plugins"
}

Hi there! Can you tell me more about your use case here, with using consul-replication for vault replication? Simply replicating the storage will not guarantee consistent results from Vault, as you've seen.

IIUC, the behaviour observed here actually counts as "Vault functioning as intended".

Vault expects and requires that it is the only thing that is writing to its storage, so by having consul-replicate write to storage that is also being used by a running Vault cluster, you create a situation where Vault will not function properly.

If you wanted to make this setup work, you would need to build a locking system so that consul-replicate, and Vault in the secondary datacenter, are never both running at the same time.

Even then, I'm not totally sure that consul-replicate would be able to preserve the ordering of operations well enough to guarantee it never places the destination Vault storage into a problematic state ... I'm thinking, what if it copies over storage entries encrypted by a new key, before the change to the keyring adding that new key?

I have some queries -

I am trying to understand the reason behind caching the nil data. one can be to prevent against DOS attack on non-existent key.

Also, I would like to understand what could possibly go wrong if we do not cache nil data as mentioned above - "you create a situation where Vault will not function properly."

"I'm thinking, what if it copies over storage entries encrypted by a new key, before the change to the keyring adding that new key?" -> are we talking about the rotation of root key scenario?

With your questions above, you've skipped past the really big important point to focus on some specific tiny parts of it, so I need to reiterate:

The Vault application is NOT written to support other software, such as consul-replicate, changing its data store underneath it whilst it is running.

You must not run consul-replicate whilst also running any Vault server in the secondary datacenter.

@maxb is correct here (thanks Max!) We cannot guarantee the results you're looking for if you change the storage. Caching, raft membership, and "eventual consistency" make this something that I don't believe we'll ever support. There are high-availability options available in Enterprise that can approximate what you're looking for (please see the page on Enterprise replication here ) but the way you're attempting to work around it will result in unexpected behavior at best and possible data corruption at worst. I'm going to close this issue now, but please feel free to open a new issue if you have further bug reports or enhancement requests. Thanks! :)

Although you're not using raft, apologies - the Integrated Storage versus Consul page makes it clear that Consul is all in-memory, which further complicates your use case here.