awskms: can't perform a seal migration, no migration seal found

[sylr]

Describe the bug

I've set up a Vault cluster on EC2 Instances which are linked to an AWS IAM instance profile which allows them to do the required AWS KMS operations on the Key I want to use for auto-unseal.

$ sudo -u vault aws sts get-caller-identity
{
    "UserId": "AROAX37UGB654AZMZXS6K:i-04fc0cb2bc4458008",
    "Account": "xxxxxxxxxxxx",
    "Arn": "arn:aws:sts::xxxxxxxxxxxx:assumed-role/CompanyEC2VaultRole/i-04fc0cb2bc4458008"
}

To Reproduce

While I try to migrate from shamir to awskms I get this:

root@vault-euc1b-dev-02:~# vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    3
Threshold                2
Version                  1.11.0
Build Date               2022-06-17T15:48:44Z
Storage Type             raft
Cluster Name             vault.dev.company.com
Cluster ID               7164c7ba-df98-12ee-9714-1bc833fbe84a
HA Enabled               true
HA Cluster               https://vault-euc1b-02.dev.company.com:8201
HA Mode                  active
Active Since             2022-07-21T07:22:34.554031132Z
Raft Committed Index     188
Raft Applied Index       188
root@vault-euc1b-dev-02:~# vault operator seal
Success! Vault is sealed.
root@vault-euc1b-dev-02:~# vault operator unseal -migrate
Unseal Key (will be hidden):
Error unsealing: Error making API request.

URL: PUT https://vault-euc1b-02.dev.company.com/v1/sys/unseal
Code: 500. Errors:

* can't perform a seal migration, no migration seal found

The only awskms related log line I could find in the server is:

Jul 21 07:22:18 vault-euc1b-dev-02 vault[65288]: 2022-07-21T07:22:18.306Z [DEBUG] core: set config: sanitized config="{\"api_addr\":\"https://vault.dev.company.com\",\"cache_size\":0,\"cluster_addr\":\"https://vault-euc1b-02.dev.company.com:8201\",\"cluster_cipher_suites\":\"\",\"cluster_name\":\"vault.dev.company.com\",\"default_lease_ttl\":0,\"default_max_request_duration\":0,\"disable_cache\":false,\"disable_clustering\":false,\"disable_indexing\":false,\"disable_mlock\":false,\"disable_performance_standby\":false,\"disable_printable_check\":false,\"disable_sealwrap\":false,\"disable_sentinel_trace\":false,\"enable_response_header_hostname\":false,\"enable_response_header_raft_node_id\":false,\"enable_ui\":true,\"listeners\":[{\"config\":{\"address\":\"0.0.0.0:443\",\"cluster_address\":\"0.0.0.0:8201\",\"tls_cert_file\":\"/etc/letsencrypt/live/vault.dev.company.com/fullchain.pem\",\"tls_key_file\":\"/etc/letsencrypt/live/vault.dev.company.com/privkey.pem\"},\"type\":\"tcp\"}],\"log_format\":\"unspecified\",\"log_level\":\"\",\"log_requests_level\":\"\",\"max_lease_ttl\":0,\"pid_file\":\"\",\"plugin_directory\":\"\",\"plugin_file_permissions\":0,\"plugin_file_uid\":0,\"raw_storage_endpoint\":false,\"seals\":[{\"disabled\":false,\"type\":\"awskms\"}],\"storage\":{\"cluster_addr\":\"https://vault-euc1b-02.dev.company.com:8201\",\"disable_clustering\":false,\"redirect_addr\":\"https://vault.dev.company.com\",\"type\":\"raft\"}}"

Expected behavior

I'd like vault to be able to use Ec2InstanceMetadata to retrieve an AWS session to be able to do the AWS KMS operations required to auto unseal Vault.

Environment:

• Vault Server Version (retrieve with vault status): 1.11.0
• Vault CLI Version (retrieve with vault version): 1.11.0
• Server Operating System/Architecture: linux/arm64

Vault server configuration file(s):

vault.hcl:

ui = true

cluster_name = "vault.dev.company.com"
api_addr = "https://vault.dev.company.com"
cluster_addr = "https://vault-euc1b-02.dev.company.com:8201"

storage "raft" {
  path = "/opt/vault/data"
  node_id = "vault-euc1b-dev-02"

  retry_join {
    leader_api_addr = "https://vault-euc1a-01.dev.company.com"
  }

  retry_join {
    leader_api_addr = "https://vault-euc1b-02.dev.company.com"
  }

  retry_join {
    leader_api_addr = "https://vault-euc1c-03.dev.company.com"
  }
}

# HTTPS listener
listener "tcp" {
  address         = "0.0.0.0:443"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file   = "/etc/letsencrypt/live/vault.dev.company.com/fullchain.pem"
  tls_key_file    = "/etc/letsencrypt/live/vault.dev.company.com/privkey.pem"
}

# Enterprise license_path
# This will be required for enterprise as of v1.8
#license_path = "/etc/vault.d/vault.hclic"

# Example AWS KMS auto unseal
seal "awskms" {
  kms_key_id = "arn:aws:kms:eu-central-1:xxxxxxxxxxxx:key/24aeeb16-4fe0-4714-997d-yyyyyyyyyyyyyy"
  region = "eu-central-1"
}

vault.env:

VAULT_SEAL_TYPE=awskms

Additional context Add any other context about the problem here.

[maxb]

You cannot perform a migration to awskms because you're already using it!

This is evidenced by the mention of "Recovery" keys in vault status, and the configuration shown in the log line you posted.

[sylr]

@maxb thank you for your time and your answer but, are you having a laugh ?

I spent six hours straight twice to debug this while there was no problem to begin with ?

I mean the Recovery Seal Type states shamir and there is absolutely no mention wheresoever that Vault is using awskms auto unseal ...

[mschultz-aofl]

Yes, this isn't well documented, but in short: When using awskms, you create a set of recovery keys. Note: These are not unseal keys - they're only used for regenerating a lost root token, and cannot under any circumstances be used to unseal vault while KMS unsealing is active. If you notice in the vault status, it says it's already unsealed. That's your key point to look for.

It says recovery seal type, as I believe under the hood, it uses the same mechanism for re-creating a root token as the shamir unsealing.

[hsimon-hashicorp]

Yes, this isn't well documented, but in short: When using awskms, you create a set of recovery keys. Note: These are not unseal keys - they're only used for regenerating a lost root token, and cannot under any circumstances be used to unseal vault while KMS unsealing is active. If you notice in the vault status, it says it's already unsealed. That's your key point to look for.

It says recovery seal type, as I believe under the hood, it uses the same mechanism for re-creating a root token as the shamir unsealing.

Yes, this is correct. Recovery keys are not the same thing as unseal keys. More on this can be found here, but it's true that it's not as clear as it could be.

[NagenderPulluri]

refer

https://support.hashicorp.com/hc/en-us/articles/10375276754707-AWS-KMS-to-AWS-KMS-Seal-Migration

From awskms to shamir:

seal "shamir" {}

seal "awskms" { region = "us-east-1" kms_key_id = "your-kms-key-id" disabled = "true" }

From shamir to awskms:

#seal "shamir" {}

seal "awskms" { region = "us-east-1" kms_key_id = "your-kms-key-id" #disabled = "true" }

[hsimon-hashicorp]

Thank you, @NagenderPulluri - I am going to go ahead and close this now, as the process is documented, if a bit roughly. Thanks everyone for your thoughts!