Initial changes for remote store cleanup during snapshot optimizations.

[harishbhakuni]

Description

• Optimize remote store cleanup to utilize batch deletion.
• Optimize snapshot deletion logic to skip only the shard blob deletion for which remote store cleanup failed.
• Maintain a local cache to track successful remote store operations which would avoid multiple calls for same resource cleanup.

Related Issues

https://github.com/opensearch-project/OpenSearch/issues/12302, https://github.com/opensearch-project/OpenSearch/issues/12253

Check List

• [ ] New functionality includes testing.
  • [ ] All tests pass
• [ ] New functionality has been documented.
  • [ ] New functionality has javadoc added
• [ ] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• [ ] Commits are signed per the DCO using --signoff
• [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• [ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

:x: Gradle check result for 79ef7924bd4e26b391c20de7c7dbc737560f3ecb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Compatibility status:

Checks if related components are compatible with change f69acff

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions[bot]]

:x: Gradle check result for a50df485690209e1e039b179dba435853caf890a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for ca5d929346c8bd5eee88efd603fd05a7f538c83d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 97c6eada40cc1ea9aaf05101d16a0348de81340f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for c48c81835784994c1e0b8379946b37027b713189: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 1bf807d93d8d3a91c0636a86f1bf93d141a8845d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for fdbf076d36daaf8cdeee4b1166f34c1baab8faf7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 81515fe4bd83af14b2b110f825ac167a20735d04: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[gbbafna]

During stale segments deletion, we are reading all the metadata files which cannot be deleted to get the list of active segment files and then comparing it with segments of the md file to be deleted. however to delete a md file X instead of reading all the md files, we can just read the md file before and after the md file X to get the segment files which cannot be deleted as part of md deletion.

I am not able to understand how will this work . Whatever the segment files were referred to by the lock files can get deleted by this approach.

[harishbhakuni]

During stale segments deletion, we are reading all the metadata files which cannot be deleted to get the list of active segment files and then comparing it with segments of the md file to be deleted. however to delete a md file X instead of reading all the md files, we can just read the md file before and after the md file X to get the segment files which cannot be deleted as part of md deletion.

I am not able to understand how will this work . Whatever the segment files were referred to by the lock files can get deleted by this approach.

apologies for the confusion, elaborated the approach in the description. basically by previous and next md file, i mean the non deletable md file before and after this md file.

[github-actions[bot]]

:x: Gradle check result for ce064fe89ed87daaa07646ef3d3ed5ccd8ec2e6c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 1b09bfcbb13c70eb9908f3f16386762b8b1a1724: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for a52fac51ce731ba16eece5bd64af96529597dbaa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 1d946bc217f8a4d5761415112916cd728de88cb7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for fcd42572e1e9c5ceecddf594ac94f1b63bb9ced5: SUCCESS

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 53.24675% with 36 lines in your changes are missing coverage. Please review.

Project coverage is 71.39%. Comparing base (b15cb0c) to head (f69acff). Report is 24 commits behind head on main.

Files	Patch %	Lines
...ch/repositories/blobstore/BlobStoreRepository.java	12.82%	33 Missing and 1 partial :warning:
...earch/index/store/RemoteSegmentStoreDirectory.java	89.47%	2 Missing :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12319      +/-   ##
============================================
- Coverage     71.42%   71.39%   -0.03%     
- Complexity    59978    59999      +21     
============================================
  Files          4985     4986       +1     
  Lines        282275   282340      +65     
  Branches      40946    40954       +8     
============================================
- Hits         201603   201575      -28     
- Misses        63999    64087      +88     
- Partials      16673    16678       +5     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[github-actions[bot]]

:x: Gradle check result for b3c2ea3630486382f37dc12d2dede40280a57ca2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for fcd42572e1e9c5ceecddf594ac94f1b63bb9ced5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for cfabef4054a0d58f14a6fe1bba7ec3c6143953c7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for fdee9ac8545b13d60d509b01399f531a0247ad0b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 93c516896042813a968e8fb3ce1744bb4507ab7c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 92a7b565ccaa470032d87a1689d35c052bfe54f1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 919e4ded85a3f24f67480c00a5c5e140540e8ee1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 1ccfa178c63f3f2a1b495b608d2a8f8f87edb671: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 47adbfea8ff4cab24934f3b9b910617ebc0b20d4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 76d1319dd9c4242d28ed377171b261625827bdeb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for 5475951520a280ecb7be483b35949aac990c3ee9: SUCCESS

[github-actions[bot]]

:white_check_mark: Gradle check result for 488c81cf97e20bf12d94ba11ebab37a26d223d65: SUCCESS

[github-actions[bot]]

:x: Gradle check result for 5ba410b10874c74c51c88e958f0ca7d59488638a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?