Introduce Interfaces to Support atomic conditional-writes
Description
This PR introduces interfaces for conditional-write support and defines new option and response types, updating the BlobContainer interface, and extending AsyncMultiStreamBlobContainer to support asynchronous conditional uploads.
-
New Types
-
ConditionalWriteOptions— an object that explicitly models each supported precondition. -
ConditionalWriteResponse— a typed result that carries provider-generated version identifiers and success status.
-
-
Conditional APIs Interfaces for
BlobContainer-
writeBlobConditionally -
writeBlobWithMetadataConditionally
-
-
Async Conditional API
-
Introduces
asyncBlobUploadConditionallytoAsyncMultiStreamBlobContainer. -
Provides the corresponding override in
AsyncMultiStreamEncryptedBlobContainerto ensure encrypted uploads honor the same contract.
-
Supported Preconditions
| Precondition | Azure | AWS S3 | GCS |
|---|---|---|---|
| If-None-Match | ETag ≠existing | If-None-Match header | — |
| If-Match | ETag == existing | If-Match header | generation match |
| If-Unmodified-Since | — | — | ifMetagenerationNotModifiedSince |
Atomic Write Workflow
-
Caller constructs a
ConditionalWriteOptionsspecifying the desired precondition. -
On success, the blob (and optional metadata) are written, returning a
ConditionalWriteResponsethat includes the new version identifier for if-match writes. -
On failure, the provider’s standard precondition-failed exception is propagated.
Note: The async implementation logic—including multipart handling via
AsyncTransferManager—is being delivered in the companion PR: https://github.com/opensearch-project/OpenSearch/pull/18093. This PR is scoped purely to defining and aligning the interfaces and types needed to support that capability.
Related Issues
-
Implements part of RFC #17763
-
Parent Meta Issue: [META] Implement Conditional APIs for Multi-Writer Detection
-
Supersedes #18065
Check List
- [x] New functionality has been documented.
- [x] Public documentation issue/PR created
- [x] Commits are signed per the DCO using --signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.
:white_check_mark: Gradle check result for 72cb9c6f7b31513467198ed41ed9a75647887b80: SUCCESS
Codecov Report
Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
Project coverage is 72.49%. Comparing base (
8948c1f) to head (99e6943).
| Files with missing lines | Patch % | Lines |
|---|---|---|
| ...org/opensearch/common/blobstore/BlobContainer.java | 0.00% | 2 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #18064 +/- ##
============================================
- Coverage 72.50% 72.49% -0.01%
+ Complexity 67167 67113 -54
============================================
Files 5473 5473
Lines 310095 310097 +2
Branches 45062 45062
============================================
- Hits 224841 224813 -28
- Misses 66899 66915 +16
- Partials 18355 18369 +14
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@x-INFiN1TY-x Please rebase your changes.
:white_check_mark: Gradle check result for 99e69436ecc6386a82e867448a48e27b18b2707c: SUCCESS
:x: Gradle check result for 6461fe02aba1159a8ab139d585f7c04bd930f443: FAILURE
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?
Flaky Test : : https://github.com/opensearch-project/OpenSearch/issues/17552
- SegmentReplicationResizeRequestIT
:x: Gradle check result for 766204dfe13788a633732a7145ae6fa87cc0f52f: FAILURE
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?
:x: Gradle check result for a928b498413b4727a59561178817ea2d88fbb23b: FAILURE
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?
Flaky Test : : https://github.com/opensearch-project/OpenSearch/issues/17937
- RethrottleTests.
can we add the required interfaces in AsyncMultiStreamBlobContainer? cc : @ashking94 @Bukhtawar @astute-decipher
Thanks for raising this, To enable asynchronous conditional uploads across providers, I’ve introduced the appropriate interface in AsyncMultiStreamBlobContainer, along with the corresponding override in AsyncMultiStreamEncryptedBlobContainer, ensuring consistent support for encrypted use cases.
This PR is specifically focused on defining and aligning the necessary interfaces to support this capability across the relevant components. The implementation details—including multipart handling login via AsyncTransferManager—are addressed in the related PR : https://github.com/opensearch-project/OpenSearch/pull/18093.
Please let me know if there’s anything further that needs to be adjusted, here's a diag detailing the intended downstream usage for the concerned interfaces. (Under review in https://github.com/opensearch-project/OpenSearch/pull/18093)
This PR is stalled because it has been open for 30 days with no activity.