Backup with transit seal method and revoked token silently fails

[laugmanuel]

Describe the bug We use Raft as our storage backend. We also do use transit sealing against a secondary Vault instance to provide auto unsealing for our primary Vault installed in Kubernetes. The token we use for that gets created by an init-container and is only valid for a few minutes. Until recently, this setup worked fine for us. The pods got unsealed automatically and the backups were present and valid (could be successfully restored).

Probably due to https://github.com/hashicorp/vault/pull/12388, this behaviour changed! Creating a backup using vault operator raft snapshot save <snapshot file> results in an error regarding the SHA256SUMS.sealed file. Using the API endpoint, we can successfully download the snapshot without any error. In both cases the snapshot file gets created and looks to contain data:

• the file size makes sense (it's not only a few bytes but matches old backups)
• using file <snapshot file> the backup is recognized as gzip compressed data

However, the backup can not be restored and Vault complains about Load error in the UI. Restoring using the CLI also fails. If I try to unpack the backup using gzip, I get unexpected end of file -> it looks like the backup file is corrupted.

If I extend the lifetime of the unseal token, the backup gets created and can be restored successfully! There is no word in the docs, that the transit token used in the Vault config/env variables must still be valid for a backup to succeed!

To Reproduce Steps to reproduce the behavior:

1. Setup a Vault with transit sealing
2. Issue a new token on the Vault providing the Transit engine
3. Unseal the new Vault using that token
4. Wait for the token to be revoked / revoke it manually
5. Create a backup using API or UI
6. Try to restore that exact backup

Expected behavior Either a valid backup file (a file that can be extracted using gzip+tar and restored) should be created; even though there is the warning about SHA256SUMS.sealed file. OR the creation of the backup should hard fail without any file being created.

If someone uses the API to create the backup but does not regularly check the restore, there would be no way to see, that the backup file is corrupted.

Also, the docs about raft snapshotting should mention, that the seal-configuration (including the token) must be valid for the backup to fully work.

Environment:

• Vault Server Version (retrieve with vault status): 1.8.3, 1.8.4
• Vault CLI Version (retrieve with vault version): 1.8.3, 1.8.4
• Server Operating System/Architecture: Linux

Vault server configuration file(s):

ui = true
disable_mlock = true
log_level = "Info"
log_format = "json"

api_addr = "http://localhost:8200"
cluster_addr = "http://localhost:8201"

listener "tcp" {
  address = "[::]:8200"
  cluster_address = "[::]:8201"

  tls_disable = 1
}

seal "transit" {
  token = "<token>" # this token is the problem
  key_name = "vault-transit"
  mount_path = "transit/"
  address = "https://transit-providing-vault:8200"
}

storage "raft" {
  path = "/data"

  retry_join {
    leader_api_addr = "http://localhost:8200"
  }
}

Additional context There must be a notice in the docs about the token used for transit. The docs and also the howto guides only mention to create a new token and to put it in the config/env variable. This would also break after the default lifetime of 32d:

• https://www.vaultproject.io/docs/commands/operator/raft#snapshot-save
• https://www.vaultproject.io/docs/configuration/seal/transit
• https://learn.hashicorp.com/tutorials/vault/autounseal-transit#step-2-configure-auto-unseal-vault-2