[WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"

[binacloud]

Describe the bug Vault app is unable to correctly parse KMS key from AWS even though roles and policies are correctly set up and KMS key is created with very permissive policy.

Storage: dynamodb (HA available)
Recovery Mode: false
Mlock: supported: true, enabled: true
Log Level: trace
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Go Version: go1.17.11
Cluster Address: https://127.0.0.1:8201
Cgo: disabled
Api Address: https://127.0.0.1:8200
==> Vault server configuration:
==> Vault server started! Log data will stream in below:
Version Sha: ea296ccf58507b25051bc0597379c467046eb2f1
Version: Vault v1.11.0, built 2022-06-17T15:48:44Z

[INFO] core: stored unseal keys supported, attempting fetch
[DEBUG] would have sent systemd notification (systemd not present): notification=READY=1
[DEBUG] cluster listener addresses synthesized: cluster_addresses=[0.0.0.0:8201]
[INFO] core: Initializing version history cache for core
[DEBUG] storage.cache: creating LRU cache: size=0
[DEBUG] core: set config: sanitized config="{\"api_addr\":\"\",\"cache_size\":0,\"cluster_addr\":\"\",\"cluster_cipher_suites\":\"\",\"cluster_name\":\"\",\"default_lease_ttl\":0,\"default_max_request_duration\":0,\"disable_cache\":false,\"disable_clustering\":false,\"disable_indexing\":false,\"disable_mlock\":false,\"disable_performance_standby\":false,\"disable_printable_check\":false,\"disable_sealwrap\":false,\"disable_sentinel_trace\":false,\"enable_response_header_hostname\":false,\"enable_response_header_raft_node_id\":false,\"enable_ui\":false,\"listeners\":[{\"config\":{\"address\":\"0.0.0.0:8200\",\"tls_cert_file\":\"/mnt/efs/tls.crt\",\"tls_disable\":\"false\",\"tls_key_file\":\"/mnt/efs/tls.key\"},\"type\":\"tcp\"}],\"log_format\":\"unspecified\",\"log_level\":\"\",\"log_requests_level\":\"\",\"max_lease_ttl\":0,\"pid_file\":\"\",\"plugin_directory\":\"\",\"plugin_file_permissions\":0,\"plugin_file_uid\":0,\"raw_storage_endpoint\":false,\"seals\":[{\"disabled\":false,\"type\":\"awskms\"}],\"storage\":{\"cluster_addr\":\"\",\"disable_clustering\":false,\"redirect_addr\":\"\",\"type\":\"dynamodb\"}}"
[INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
[WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
[INFO] core: stored unseal keys supported, attempting fetch
[WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
[INFO] core: stored unseal keys supported, attempting fetch
[WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
[INFO] core: stored unseal keys supported, attempting fetch
[WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
[INFO] core: stored unseal keys supported, attempting fetch
[WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
[INFO] core: stored unseal keys supported, attempting fetch
[WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"

To Reproduce I'm deploying using Terraform, the latest Docker image Hashicorp Vault 1.11.0 on Amazon ECS, using DynamoDB as the backend.

Environment variables declared in container_definitions :

VAULT_ADDR=https://0.0.0.0:8201/
VAULT_API_ADDR=https://127.0.0.1:8201
VAULT_SEAL_TYPE=awskms
VAULT_AWSKMS_SEAL_KEY_ID=arn:aws:kms:<REGION>:<ACCOUNT_ID>:key/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
VAULT_UI=true
VAULT_LOCAL_CONFIG={storage: {dynamodb: {ha_enabled: true, region: ${var.region}, table: ${aws_dynamodb_table.vault_dynamo.name} }}, listener: {tcp: { address: 0.0.0.0:8201, tls_disable: false, tls_cert_file: /mnt/efs/tls.crt, tls_key_file: /mnt/efs/tls.key } } }

Certificate and private key are mounted as EFS volume.

Expected behavior

• KMS key to be read correctly by Vault application, and used for initial Vault un-seal.
• In case of errors, log out something more specific.
• When we set log level to "trace", we really do expect more details. There seems to be no difference between "info" and "trace" for this situation above.

Environment:

• Vault Server Version (retrieve with vault status): 1.11.0
• Vault CLI Version (retrieve with vault version):
• Server Operating System/Architecture: Vault's official Docker image dpeloyed on AWS ECS

Vault server configuration file(s):

# As above

Additional context The KMS key definitely exist and correct. When I try a false KMS key ID, it logs "key not found". When I try a KMS key ID from another AWS account, it logs "Access Denied"

The IAM policy of the KMS key is as permissive as possible, and also the ECS assumed task role is super permissive. Even if it's not I should see some permission or access related errors right?? So why is it giving this vague warning that's clearly an error ??

I've enabled "trace" level log and still nothing useful. This should be marked as [ERROR] because it fails the container after some time.

What does this error mean ??

[maxb]

The comment in the code explains more:

https://github.com/hashicorp/vault/blob/27f15edd9f01cb64977a76c2d72de0655b243808/vault/init.go#L466-L470

[sgmiller]

Right, I think you just haven't initialized with vault operator init yet.

[markmartirosian]

I see this after vault operator init on the other nodes, until they are restarted, then everything is as expected.

[vini-gorillas]

After i do a vault operator init the recovery seal type changes to shamiris this the expected behaviour? How. does the key ring work in case the vault is sealed? How can i use the key ring for the unseal?

[smrutimandal]

I see this after vault operator init on the other nodes, until they are restarted, then everything is as expected.

I deploy a 3 node HA cluster to K8s and even after one of the nodes has initialized the cluster, I get the following error in the other two nodes

{"@level":"info","@message":"stored unseal keys supported, attempting fetch","@module":"core","@timestamp":"2022-12-05T09:59:56.347265Z"}
{"@level":"warn","@message":"failed to unseal core","@timestamp":"2022-12-05T09:59:56.347377Z","error":"stored unseal keys are supported, but none were found"}
{"@level":"info","@message":"security barrier not initialized","@module":"core","@timestamp":"2022-12-05T10:00:00.640206Z"}
{"@level":"info","@message":"seal configuration missing, but cannot check old path as core is sealed","@module":"core.autoseal","@timestamp":"2022-12-05T10:00:00.640260Z","seal_type":"recovery"}

As a work around, I am force restarting(by killing the two pods) them and then all pods are healthy.

[a14m]

@smrutimandal I was running into the same issue... but after adding the following 2 configurations to the helm chart

server:
  # yada yada
  readinessProbe:
    enabled: true
  livenessProbe:
    enabled: true
  #yada yada

(with the default settings in the helm chart)... K8s will automatically restart the unhealthy pods and they will work afterwords (instead of manually force restarting).

Ref to the helm chart configs