:zap: Add support for high availability S3 backend

[danquack]

Description

This PR prototypes state locking for Vault's s3 backend using Amazon S3’s recently introduced conditional writes feature to implement a locking mechanism.

Currently S3 does not implement the physical.HABackend to support high availability. With S3 Conditional writes, the if-none-match header can be applied during the Lock PutObject, which will enforce a remote lock state. In order to introduce this, an upgrade from sdk v1 to sdkv2 was needed. As a result, some work toward #18375 could be achieved due to how the configuration is loaded. This only applies to storage, as additional work would be needed to fully close that issue.

Demo

A quick demo to show two servers, where any writes to the first server can be read by the second, and any writes to the second can be read by the first.

Tests

# go test -v ./physical/s3/...                                                                                                                                  
=== RUN   TestHABackend
2024-12-13T16:15:22.108-0500 [DEBUG] using other creds
2024-12-13T16:15:22.225-0500 [DEBUG] max_parallel set: max_parallel=0
2024-12-13T16:15:22.225-0500 [DEBUG] using other creds
2024-12-13T16:15:22.343-0500 [DEBUG] max_parallel set: max_parallel=0
--- PASS: TestHABackend (12.90s)
=== RUN   TestDefaultS3Backend
2024-12-13T16:15:34.729-0500 [DEBUG] added environment variable credential provider
2024-12-13T16:15:34.729-0500 [DEBUG] added shared credential provider
2024-12-13T16:15:35.061-0500 [DEBUG] using other creds
2024-12-13T16:15:35.247-0500 [DEBUG] max_parallel set: max_parallel=0
--- PASS: TestDefaultS3Backend (2.65s)
=== RUN   TestS3BackendSseKms
2024-12-13T16:15:37.376-0500 [DEBUG] added environment variable credential provider
2024-12-13T16:15:37.376-0500 [DEBUG] added shared credential provider
2024-12-13T16:15:37.705-0500 [DEBUG] using other creds
2024-12-13T16:15:37.863-0500 [DEBUG] max_parallel set: max_parallel=0
--- PASS: TestS3BackendSseKms (2.30s)
PASS
ok  	github.com/hashicorp/vault/physical/s3	18.084s

TODO only if you're a HashiCorp employee

• [ ] Backport Labels: If this fix needs to be backported, use the appropriate backport/ label that matches the desired release branch. Note that in the CE repo, the latest release branch will look like backport/x.x.x, but older release branches will be backport/ent/x.x.x+ent.
  • [ ] LTS: If this fixes a critical security vulnerability or severity 1 bug, it will also need to be backported to the current LTS versions of Vault. To ensure this, use all available enterprise labels.
• [ ] ENT Breakage: If this PR either 1) removes a public function OR 2) changes the signature of a public function, even if that change is in a CE file, double check that applying the patch for this PR to the ENT repo and running tests doesn't break any tests. Sometimes ENT only tests rely on public functions in CE files.
• [ ] Jira: If this change has an associated Jira, it's referenced either in the PR description, commit message, or branch name.
• [ ] RFC: If this change has an associated RFC, please link it in the description.
• [ ] ENT PR: If this change has an associated ENT PR, please link it in the description. Also, make sure the changelog is in this PR, not in your ENT PR.

[hashicorp-cla-app[bot]]

All committers have signed the CLA.

[schavis]

We are migrating Vault content to a different repo. Please recreate the content portion of this PR against the hashicorp/web-unified-docs repo on or after Monday, July 21, 2025.

[gerlamberti]

Is there any news regarding the possible implementation of this feature in upcoming Vault versions?