Upload translog.ckp file data as object metadata with translog.tlog file for s3 remote store

[skumawat2025]

Description

This PR is a sub task of feature request https://github.com/opensearch-project/OpenSearch/issues/13022. And is being tracked here https://github.com/opensearch-project/OpenSearch/issues/13091.

This PR include below changes -

1. The translog.ckp file's data is now stored as object metadata associated with the translog.tlog file when uploading to an S3 remote storage.
2. The translog download flow has been updated to handle mixed index scenarios. In cases where translog files were previously uploaded to the remote storage in different formats (separate translog.tlog and translog.ckp files) as well as the new upload format (checkpoint file as header/metadata to translog.tlog file)
3. Deletion of translog files

Related Issues

Resolves https://github.com/opensearch-project/OpenSearch/issues/13094

Check List

• [x] New functionality includes testing.
  • [x] All tests pass
• [x] New functionality has been documented.
  • [x] New functionality has javadoc added
• [x] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• [x] Commits are signed per the DCO using --signoff
• [ ] ~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~
• [ ] ~Public documentation issue/PR created~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

:x: Gradle check result for b22a58d973a9ded402cb158242d6f8d3e85c91f9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for fde978830857d27748a840581955c7baf5d56db5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 57326eb9f33b283081ffddcebf720dfd01928bdb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 23f4a64d9dbc2c30e267786f078fb49c05b39146: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for e69b2b42d2ff90f92c3f6af2444226ecdb662a3c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 5dc4ab27131f85c5d5ae1335de8b5c5da824da0b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 85.23316% with 57 lines in your changes are missing coverage. Please review.

Project coverage is 71.55%. Comparing base (b15cb0c) to head (bd58d92). Report is 275 commits behind head on main.

:exclamation: Current head bd58d92 differs from pull request most recent head fd5450f. Consider uploading reports for the commit fd5450f to get more accurate results

Files	Patch %	Lines
...sfer/TranslogCkpAsMetadataFileTransferManager.java	86.76%	4 Missing and 5 partials :warning:
.../translog/transfer/TranslogCheckpointSnapshot.java	76.47%	8 Missing :warning:
...slog/transfer/TranslogCkpFilesTransferManager.java	87.27%	1 Missing and 6 partials :warning:
...slog/transfer/TranslogCkpFilesTransferTracker.java	88.88%	3 Missing and 4 partials :warning:
.../org/opensearch/index/remote/RemoteStoreUtils.java	50.00%	0 Missing and 4 partials :warning:
...ex/translog/transfer/BlobStoreTransferService.java	20.00%	4 Missing :warning:
...ex/remote/RemoteMigrationIndexMetadataUpdater.java	62.50%	0 Missing and 3 partials :warning:
...ensearch/index/translog/transfer/FileSnapshot.java	66.66%	3 Missing :warning:
...rg/opensearch/index/translog/RemoteFsTranslog.java	83.33%	1 Missing and 1 partial :warning:
...va/org/opensearch/indices/RemoteStoreSettings.java	66.66%	2 Missing :warning:
... and 8 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13272      +/-   ##
============================================
+ Coverage     71.42%   71.55%   +0.13%     
- Complexity    59978    61172    +1194     
============================================
  Files          4985     5060      +75     
  Lines        282275   287403    +5128     
  Branches      40946    41628     +682     
============================================
+ Hits         201603   205650    +4047     
- Misses        63999    64830     +831     
- Partials      16673    16923     +250     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[github-actions[bot]]

:grey_exclamation: Gradle check result for b64d2d911c17fc69fd9777d7a0074d452a55764e: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions[bot]]

:x: Gradle check result for a9223ad15a1cd4e5a47249f1190aa2a13d5edd56: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[skumawat2025]

❌ Gradle check result for a9223ad: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

flaky: https://github.com/opensearch-project/OpenSearch/issues/12197

[github-actions[bot]]

:white_check_mark: Gradle check result for 23c8673fffb8d0960477d51863e120f95a4966ed: SUCCESS

[github-actions[bot]]

:white_check_mark: Gradle check result for 6fac833c95fcd8e1e2555100fc40638e541984d6: SUCCESS

[github-actions[bot]]

:x: Gradle check result for df169f0178e05c7208e413827d007e4c3adb98a9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for dee19299e867b72c64c0f8e7248a446110acefcb: SUCCESS

[github-actions[bot]]

:x: Gradle check result for 17928ddaafd260963fa2f5b085867acbea2c1ff9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 0a8010d1c9bc79fb810f10d5b4daf1615129c95d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 5368109011695c318d32829f7d1c81266d37ddc3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 80b29e13370fce2e6332a8e2ad3d668a3b70eb83: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for e0c2bfd678c02325cd6637cb4355e817750890aa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for f4a39add24ed5cdc12dbd20354cb6215a7f9523d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 23ab7a668e1decb78fc4af46c8315dffc0c93cfd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 61d84b58ad9dafb6e91acf5a9f978be53226db07: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 7d0c07e016bb4acb4df3f4ce21b98af3ef6348e0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for b174d68b33b56429e6c5268f8b5a5cd54091ce55: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for c592b7207a39570d8aef31ea5e7315a00a18c65f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for ecb32d6441bb52ca2b90d811b9ad67abd9636ec0: SUCCESS

[github-actions[bot]]

:x: Gradle check result for f1ceecc85110cab042f3db5686f6249c2489638b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for d49c8901a29fa0338ea58e67c95514aade9760d7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for f01db951fce4e38353dd75a4b62254e64415e0a0: TIMEOUT

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[linuxpi]

[Storage Triage - attendees 1 2 3 4 5 6 7 8 9 10 11 12 13]

Since 2.14 is already cut, moving this to 2.15 release target. @skumawat2025 looking forward seeing this improvement in 2.15