The core/lock file is not created in the GCS bucket after upgrading to version 1.20.0

[verdel]

Describe the bug After upgrading Vault from version 1.19.5 to 1.20.0, using a GCS bucket as the storage backend with the ha_enabled=true option enabled, the following error appears in the pod:

core: failed to acquire lock: error="lock: attempt lock: write lock: failed to read attrs for \"core/lock\": storage: object doesn't exist: googleapi: Error 404: No such object: <gcs_bucket_name>/core/lock, notFound"

The core/lock file does not appear in the bucket itself:

> gsutil ls gs://<gcs_bucket_name>/core

gs://<gcs_bucket_name>/core/audit
gs://<gcs_bucket_name>/core/auth
gs://<gcs_bucket_name>/core/canary-keyring
gs://<gcs_bucket_name>/core/index-header-hmac-key
gs://<gcs_bucket_name>/core/keyring
gs://<gcs_bucket_name>/core/local-audit
gs://<gcs_bucket_name>/core/local-auth
gs://<gcs_bucket_name>/core/local-mounts
gs://<gcs_bucket_name>/core/master
gs://<gcs_bucket_name>/core/mounts
gs://<gcs_bucket_name>/core/recovery-config
gs://<gcs_bucket_name>/core/recovery-key
gs://<gcs_bucket_name>/core/recovery-keys-backup
gs://<gcs_bucket_name>/core/seal-config
gs://<gcs_bucket_name>/core/seal-gen-info
gs://<gcs_bucket_name>/core/shamir-kek
gs://<gcs_bucket_name>/core/cluster/
gs://<gcs_bucket_name>/core/hsm/
gs://<gcs_bucket_name>/core/plugin-catalog/
gs://<gcs_bucket_name>/core/versions/
gs://<gcs_bucket_name>/core/wrapping/

To Reproduce

1. Install Vault version 1.19.5 in the Kubernetes cluster using the official Helm chart
2. Use a GCS bucket as the storage backend
3. Update the Vault Docker image version to 1.20.0
4. Check the events in the Vault pod
5. Check for the existence of the core/lock file in the GCS bucket

Expected behavior Upgrading to version 1.20.0 without changing the storage configuration should not result in an error creating the lock file or prevent the Vault cluster from starting.

Environment:

• Vault Server Version (retrieve with vault status): 1.20.0
• Vault CLI Version (retrieve with vault version): v1.20.0 (6fdd6b59e97d97a9e19b0fb5304bf879c190295e), built 2025-06-23T10:21:30Z
• Server Operating System/Architecture: GKE Kubernetes cluster v1.30.5-gke.1014003

Vault server configuration file(s):

ui = true

listener "tcp" {
  tls_cert_file = "/vault/userconfig/vault-tls/tls.crt"
  tls_key_file = "/vault/userconfig/vault-tls/tls.key"
  address = "[::]:8200"
  cluster_address = "[::]:8201"
}

plugin_directory = "/usr/local/libexec/vault"

storage "gcs" {
  bucket = "<gcs_bucket_name>"
  ha_enabled = "true"
  chunk_size = "512"
}

service_registration "kubernetes" {}

seal "awskms" {
  region     = "<region>"
  kms_key_id = "<kms_key_id>"
}
disable_mlock = true

Additional context

[verdel]

I tried creating a clean installation of Vault version 1.19.5 with the configuration mentioned above. Then I changed the Docker tag to 1.20.0 and encountered the same error.

After that, I tried creating a clean installation of Vault version 1.20.0, and the error still persists.

[fancybear-dev]

Can confirm this is an issue for us as well. We're unable to generate tokens or perform API requests due to this bug - i.e. Vault is essentially unusable in this state. We do not use this backend for production environments, given the lack of official support. We reverted to last known good, 1.19.5.

[thidajat-prosper]

Confirmed we are seeing this issue as well, after node pool rotation vault was not able to bootstrap and seeing core/lock error. Pinning version to 1.19 fix the issue.

[foppepieters]

Same issue here. After renovate tested an upgrade it failed and vault was in the same broken state with:

[ERROR] core: failed to acquire lock: error="lock: attempt lock: write lock: failed to read attrs for \"core/lock\": storage: object doesn't exist: googleapi: Error 404: No such object

We are also pinning the version to 1.19.5

[paul-at-cybr]

The default version in the helmchart recently got bumped to 1.20.1. I suspect this issue is about to get a bit more attention.

[verdel]

Yes, unfortunately, there hasn’t been a release yet that includes the fixes from the PR that was merged into the main branch.

[paul-at-cybr]

The fix is included in 1.20.2, and while the statefulset no longer crashloops, it still doesn't create a lock file. Is there a recipe for a manual intervention anywhere?

[gysel]

We are also affected by this.

[kathleenfrench]

The fix is included in 1.20.2, and while the statefulset no longer crashloops, it still doesn't create a lock file. Is there a recipe for a manual intervention anywhere?

i don't think the fix is included in 1.20.2

[verdel]

@paul-at-cybr, in version 1.20.2, the fixes have also not yet been added.

[xueshanf]

Any workaround? Is it safe to downgrade? I tried to touch a lock file in the GCS bucket location, but it gives different error (json file error).

[gysel]

I did the downgrade from 1.20.2 to 1.19.5 and it seems to be working fine so far.

[froblesmartin]

For those ending up here, if you are using the official Helm chart, you can just add this in your values to pin to the previous version of Vault:

server:
  image:
    tag: "1.19.5"

[xueshanf]

I rolled back to 1.19.5 as well (had to remove the lock file I manually created). It seems working fine.

[heinrichgrt]

Same here 1.20.0 and 1.20.1 are not working with gcp storage buckets.

[soumodeep46]

Facing issue with 1.20.1 and 1.20.2 versions as they are not working with gcp storage buckets. Please help for a fix as <1.20.0 versions have security vulnerabilities

[verdel]

The fix was also not included in the 1.20.3 release.

[stevemsmith]

It looks like #31274 was merged, will the fix be available in 1.20.4?

[soumodeep46]

Still facing this issue even after using 1.20.4

[sapk]

https://github.com/hashicorp/vault/pull/31274 is merged in next version branch (1.21.X). I backported the really simple fix in support branch of 1.20.X but it is not merged yet https://github.com/hashicorp/vault/pull/31525.

[9digitdev]

Is there anything the community can do to help move this along? It would be nice to get this bug fixed.

[minivendra]

Another one impacted here, please try to have a patch ready for 1.20.x thanks!

[soumodeep46]

Is there any update on this issue? I think it's quite a long time we are facing this issue

[funzie19]

Any updates on this issue?

[xueshanf]

Looks like the code fix was merged in 1.21.0-rc code but not mentioned in its release notes.

[paul-at-cybr]

Confirmed - the vault image 1.21.0-rc1 works with the 0.31.0 helmchart.

[xueshanf]

Confirmed - the vault image 1.21.0-rc1 works with the 0.31.0 helmchart.

Are you able to go from 1.19.x to 1.21? 1.21.0 is releases a few hour ago.

[drone2of6]

Confirmed - the vault image 1.21.0-rc1 works with the 0.31.0 helmchart.

Are you able to go from 1.19.x to 1.21? 1.21.0 is releases a few hour ago.

I just upgraded through dnf (not helm), but went from 1.19.x to the latest 1.21.0 and it's worked fine.

[xueshanf]

Yay! Thank you for verifying.

[verdel]

Yes, I confirm that the issue is no longer present in version 1.21.0.