OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

Introduce Interfaces to Support atomic conditional-writes

Open x-INFiN1TY-x opened this issue 8 months ago • 10 comments

Description

This PR introduces interfaces for conditional-write support and defines new option and response types, updating the BlobContainer interface, and extending AsyncMultiStreamBlobContainer to support asynchronous conditional uploads.

  • New Types

    • ConditionalWriteOptions — an object that explicitly models each supported precondition.

    • ConditionalWriteResponse — a typed result that carries provider-generated version identifiers and success status.

  • Conditional APIs Interfaces for BlobContainer

    • writeBlobConditionally

    • writeBlobWithMetadataConditionally

  • Async Conditional API

    • Introduces asyncBlobUploadConditionally to AsyncMultiStreamBlobContainer.

    • Provides the corresponding override in AsyncMultiStreamEncryptedBlobContainer to ensure encrypted uploads honor the same contract.

Supported Preconditions

Precondition Azure AWS S3 GCS
If-None-Match ETag ≠ existing If-None-Match header —
If-Match ETag == existing If-Match header generation match
If-Unmodified-Since — — ifMetagenerationNotModifiedSince

Atomic Write Workflow

  1. Caller constructs a ConditionalWriteOptions specifying the desired precondition.

  2. On success, the blob (and optional metadata) are written, returning a ConditionalWriteResponse that includes the new version identifier for if-match writes.

  3. On failure, the provider’s standard precondition-failed exception is propagated.

Note: The async implementation logic—including multipart handling via AsyncTransferManager—is being delivered in the companion PR: https://github.com/opensearch-project/OpenSearch/pull/18093. This PR is scoped purely to defining and aligning the interfaces and types needed to support that capability.

Related Issues

Check List

  • [x] New functionality has been documented.
  • [x] Public documentation issue/PR created
  • [x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

x-INFiN1TY-x avatar Apr 24 '25 01:04 x-INFiN1TY-x

:white_check_mark: Gradle check result for 72cb9c6f7b31513467198ed41ed9a75647887b80: SUCCESS

github-actions[bot] avatar Apr 24 '25 03:04 github-actions[bot]

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 72.49%. Comparing base (8948c1f) to head (99e6943).

Files with missing lines Patch % Lines
...org/opensearch/common/blobstore/BlobContainer.java 0.00% 2 Missing :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #18064      +/-   ##
============================================
- Coverage     72.50%   72.49%   -0.01%     
+ Complexity    67167    67113      -54     
============================================
  Files          5473     5473              
  Lines        310095   310097       +2     
  Branches      45062    45062              
============================================
- Hits         224841   224813      -28     
- Misses        66899    66915      +16     
- Partials      18355    18369      +14     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Apr 24 '25 03:04 codecov[bot]

@x-INFiN1TY-x Please rebase your changes.

skumawat2025 avatar Apr 24 '25 10:04 skumawat2025

:white_check_mark: Gradle check result for 99e69436ecc6386a82e867448a48e27b18b2707c: SUCCESS

github-actions[bot] avatar Apr 24 '25 22:04 github-actions[bot]

:x: Gradle check result for 6461fe02aba1159a8ab139d585f7c04bd930f443: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar Apr 27 '25 22:04 github-actions[bot]

Flaky Test : : https://github.com/opensearch-project/OpenSearch/issues/17552

  • SegmentReplicationResizeRequestIT

x-INFiN1TY-x avatar Apr 27 '25 22:04 x-INFiN1TY-x

:x: Gradle check result for 766204dfe13788a633732a7145ae6fa87cc0f52f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar May 28 '25 13:05 github-actions[bot]

:x: Gradle check result for a928b498413b4727a59561178817ea2d88fbb23b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar May 28 '25 14:05 github-actions[bot]

Flaky Test : : https://github.com/opensearch-project/OpenSearch/issues/17937

  • RethrottleTests.

x-INFiN1TY-x avatar May 28 '25 14:05 x-INFiN1TY-x

can we add the required interfaces in AsyncMultiStreamBlobContainer? cc : @ashking94 @Bukhtawar @astute-decipher

Thanks for raising this, To enable asynchronous conditional uploads across providers, I’ve introduced the appropriate interface in AsyncMultiStreamBlobContainer, along with the corresponding override in AsyncMultiStreamEncryptedBlobContainer, ensuring consistent support for encrypted use cases.

This PR is specifically focused on defining and aligning the necessary interfaces to support this capability across the relevant components. The implementation details—including multipart handling login via AsyncTransferManager—are addressed in the related PR : https://github.com/opensearch-project/OpenSearch/pull/18093.

Please let me know if there’s anything further that needs to be adjusted, here's a diag detailing the intended downstream usage for the concerned interfaces. (Under review in https://github.com/opensearch-project/OpenSearch/pull/18093) Mermaid Chart - Create complex, visual diagrams with text  A smarter way of creating diagrams -2025-05-28-172238

x-INFiN1TY-x avatar May 28 '25 17:05 x-INFiN1TY-x

This PR is stalled because it has been open for 30 days with no activity.