OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

Introduce & Implement Support for Versioning Operations in BlobContainer & S3BlobContainer

Open x-INFiN1TY-x opened this issue 7 months ago • 6 comments

Introduce & Implement Support for Versioning Operations in BlobContainer & S3BlobContainer

This PR introduces new interfaces in the BlobContainer abstraction to support versioned blob operations and implements them in the S3BlobContainer. It enables clients to list all versions of a blob, retrieve a specific version (optionally with byte range), and access version metadata—both synchronously and asynchronously


At a Glance

Method Location Purpose
listBlobVersions(String blobName) Interface: BlobContainer List all versions of a blob (sync)
listBlobVersionsSorted(String blobName) Interface: BlobContainer List versions sorted by timestamp
listBlobVersions(String, ActionListener<...>) Interface: BlobContainer List all versions (async)
listBlobVersionsSorted(String, ActionListener<...>) Interface: BlobContainer List sorted versions (async)
readBlobVersion(String blobName, String versionId) Interface: BlobContainer Retrieve a specific version (sync)
readBlobVersion(String, String, long, long) Interface: BlobContainer Retrieve a version range (sync)
readBlobVersion(String, String, ActionListener<...>) Interface: BlobContainer Retrieve a specific version (async)
headBlobVersion(String blobName, String versionId) Interface: BlobContainer Fetch version metadata (sync)
headBlobVersion(String, String, ActionListener<...>) Interface: BlobContainer Fetch version metadata (async)
Implemented in S3BlobContainer S3-specific versioning operations

Summary of Changes

  • Implement listBlobVersions(...) and listBlobVersionsSorted(...) in S3BlobContainer using S3’s ListObjectVersionsRequest, filtering out delete markers and mapping to BlobMetadata.

  • Implement readBlobVersion(...) (full or ranged) via S3’s GetObjectRequest(versionId[, range]).

  • Implement headBlobVersion(...) via S3’s HeadObjectRequest(versionId), returning BlobMetadata.

  • Provide asynchronous wrappers for all new methods using ActionListener.

  • Handle errors:

    • Throw BlobStoreException if versioning is disabled.

    • Translate S3 404 to NoSuchFileException.

    • Wrap other S3/SDK errors in IOException.


Core Implementation

  1. listBlobVersions(String blobName)

    • Paginate ListObjectVersionsRequest(bucket, prefix=key).

    • Filter out delete markers and map each ObjectVersion to a PlainBlobMetadata (with versionId, sanitized eTag, and lastModified).

    • Throw BlobStoreException if versioning is disabled; NoSuchFileException if bucket is missing; wrap others in IOException.

  2. listBlobVersionsSorted(String blobName)

    • Delegate to listBlobVersions(...) and sort versions by descending lastModified.

  3. readBlobVersion(...)

    • Validate versionId (non-null).

    • Build and execute GetObjectRequest(bucket, key, versionId[, range]).

    • Return the resulting InputStream; on 404, throw NoSuchFileException; wrap other errors in IOException.

  4. headBlobVersion(String blobName, String versionId)

    • Validate versionId (non-null).

    • Build and execute HeadObjectRequest(bucket, key, versionId).

    • Map response to a PlainBlobMetadata (handling null eTag or lastModified).

    • On 404, throw NoSuchFileException; wrap others in IOException.

  5. Asynchronous Variants

    • Each default async method calls its sync counterpart in a try/catch, invoking listener.onResponse(...) on success or listener.onFailure(e) on exception.


Test Coverage Highlights

  • testRepositoryVersionOperations

    • Verifies non-null results and correct sorting for listBlobVersions(...) / listBlobVersionsSorted(...) when versioning is enabled; checks for clear exceptions when versioning is unsupported.

  • testVersionParameterValidation

    • Asserts that null versionId and negative position/length arguments throw IllegalArgumentException.

  • testListBlobVersionsOnlyProcessesVersionsNotDeleteMarkers

    • Confirms delete markers are excluded.

  • testListBlobVersionsExactKeyMatch

    • Ensures only exact-key matches are returned, not prefixes or suffixes.

  • testReadBlobVersionRangeBehavior

    • Validates correct Range header logic for non-zero length, empty stream for length == 0, and input validations.

  • testNullSafetyInMetadataCreation

    • Mocks null eTag/lastModified and asserts resulting BlobMetadata uses "" and 0 respectively.

Related Issues

  • Implements part of [RFC #17763](https://github.com/opensearch-project/OpenSearch/issues/17763)
  • Parent Meta Issue: [#17859](https://github.com/opensearch-project/OpenSearch/issues/17859)

Check List

  • [ ] Functionality includes testing.
  • [ ] API changes companion pull request created, if applicable.
  • [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

x-INFiN1TY-x avatar Jun 05 '25 13:06 x-INFiN1TY-x

@ashking94 @astute-decipher Could you please review this PR.

x-INFiN1TY-x avatar Jun 05 '25 13:06 x-INFiN1TY-x

:x: Gradle check result for 4e26b67833105e3914559b263a6d02fa1e7ac4a6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar Jun 05 '25 13:06 github-actions[bot]

:white_check_mark: Gradle check result for ea67f98c960c32685eb415234dae18ab97822b70: SUCCESS

github-actions[bot] avatar Jun 06 '25 11:06 github-actions[bot]

Codecov Report

Attention: Patch coverage is 41.05960% with 89 lines in your changes missing coverage. Please review.

Project coverage is 72.69%. Comparing base (e417086) to head (ea67f98). Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
...rg/opensearch/repositories/s3/S3BlobContainer.java 57.14% 39 Missing and 6 partials :warning:
...org/opensearch/common/blobstore/BlobContainer.java 7.40% 25 Missing :warning:
...earch/common/blobstore/EncryptedBlobContainer.java 0.00% 16 Missing :warning:
.../org/opensearch/common/blobstore/BlobMetadata.java 0.00% 3 Missing :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #18448      +/-   ##
============================================
- Coverage     72.69%   72.69%   -0.01%     
- Complexity    67823    67878      +55     
============================================
  Files          5516     5517       +1     
  Lines        312444   312595     +151     
  Branches      45359    45379      +20     
============================================
+ Hits         227127   227227     +100     
- Misses        66768    66843      +75     
+ Partials      18549    18525      -24     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Jun 06 '25 11:06 codecov[bot]

:x: Gradle check result for fc10bcacce1bd9f7ad2b87a8ac49b4741d77e934: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar Jun 14 '25 09:06 github-actions[bot]

:x: Gradle check result for ebd166b560bce22f1bd7a320b5d10f55a8b8d0e8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar Jun 14 '25 14:06 github-actions[bot]

This PR is stalled because it has been open for 30 days with no activity.