[Writable Warm] Composite Directory implementation and integrating it with FileCache

[rayshrey]

Description

Implements and design Composite Directory (and integrates it with FileCache) which can fall back to the on-demand fetch when requested data is not present locally.

Opening this PR to get initial review comments on the design and implementation of Composite Directory in this issue - https://github.com/opensearch-project/OpenSearch/issues/12781, hence haven't added any tests apart from one IT test for testing the basic functionality. Will include tests in the following revisions to this PR.

Related Issues

Resolves #[12781]

Check List

• [x] New functionality includes testing.
  • [x] All tests pass
• [x] New functionality has been documented.
  • [x] New functionality has javadoc added
• [x] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• [x] Commits are signed per the DCO using --signoff
• [x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• [ ] ~Public documentation issue/PR created~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

Compatibility status:

Checks if related components are compatible with change f442533

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git]

[github-actions[bot]]

:x: Gradle check result for d50732721ad5f22a317aec3ee49762b606ce0bef: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for e9ec1d4681c039450145968d6d0042a2bbd6282d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for d1b63f0ae2e9f65b36dd9efa7fd74bdec539b410: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 1efe68fcb9ab80f5e64a6fce3325532c01045d17: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 2854383bbf81dbf7be9321caf5dc5af7b36cf47b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 958d4f588e2b42b484fe4513ccc35fe34fd4ba24: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for f44253392639a30ed070f82bf9f681b43d137ac2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 70bac1fd8bb04337bb26f738ddb620bb63322f86:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 8430bf11f6467cb95d4e5e68b56ad75bf6a7156d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 8cbfeead8c76e802a91747b8c55ed5bae0d7fab1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for ca1060f3face7b354046635d8f78d983ebd123f0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 3d92046be3ddf6c3572defe61fee3bb20d72888c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:grey_exclamation: Gradle check result for 93d1f2a76a0a3addcd58d1f99411d1d4bb215891: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringQueryPhase

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 75.17241% with 72 lines in your changes missing coverage. Please review.

Project coverage is 71.74%. Comparing base (b15cb0c) to head (901849b). Report is 459 commits behind head on main.

Files	Patch %	Lines
...org/opensearch/index/store/CompositeDirectory.java	81.48%	13 Missing and 7 partials :warning:
...earch/index/store/remote/utils/cache/LRUCache.java	33.33%	10 Missing :warning:
...c/main/java/org/opensearch/index/IndexService.java	36.36%	6 Missing and 1 partial :warning:
...index/store/remote/utils/cache/SegmentedCache.java	14.28%	6 Missing :warning:
...search/index/store/remote/filecache/FileCache.java	16.66%	5 Missing :warning:
...ore/remote/filecache/FullFileCachedIndexInput.java	85.71%	4 Missing and 1 partial :warning:
...search/index/store/remote/utils/FileTypeUtils.java	33.33%	4 Missing :warning:
server/src/main/java/org/opensearch/node/Node.java	50.00%	0 Missing and 3 partials :warning:
...rc/main/java/org/opensearch/index/IndexModule.java	89.47%	1 Missing and 1 partial :warning:
...search/index/shard/RemoteStoreRefreshListener.java	33.33%	1 Missing and 1 partial :warning:
... and 6 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12782      +/-   ##
============================================
+ Coverage     71.42%   71.74%   +0.32%     
- Complexity    59978    62202    +2224     
============================================
  Files          4985     5124     +139     
  Lines        282275   292248    +9973     
  Branches      40946    42233    +1287     
============================================
+ Hits         201603   209687    +8084     
- Misses        63999    65327    +1328     
- Partials      16673    17234     +561     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[github-actions[bot]]

:x: Gradle check result for 3280a6db5e2e0eb15cb6ffadcae6e561f1c33fbe: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[rayshrey]

Build failing due to unrelated Flaky tests:

https://github.com/opensearch-project/OpenSearch/issues/13473 https://github.com/opensearch-project/OpenSearch/issues/13600

[github-actions[bot]]

:white_check_mark: Gradle check result for 4ee744d9103c7764c213da3bf85983c1964168ff: SUCCESS

[github-actions[bot]]

:x: Gradle check result for 3292b36690ec9c619bee734b42e8e43961b3ce34: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for 5013be961c7f32f06ade59dd802a4e7b46ec48e3: SUCCESS

[github-actions[bot]]

:x: Gradle check result for 0773fdd3a618bce9189699e5a1314fef9f07ab68: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for b6eee23f7eb122c89093bdaaf82530826b250dfc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 7491c62643b44760824ab82d6e117fb3c0c5609c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 78fee6eedfe4e05950c2cea62e282ca1d92d19a0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for fd20fad0f1a28e91fd4d5ffb89312d5e10adfdbf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 87eed3c1118db98afc6e504d4b22790040b13fe9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[rayshrey]

Failure related to flaky test - https://github.com/opensearch-project/OpenSearch/issues/10735

[github-actions[bot]]

:x: Gradle check result for 61849d6b39d9031da0a58c3d407e00bb0404cffc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[rayshrey]

Failure due to Flaky test - https://github.com/opensearch-project/OpenSearch/issues/13890

[github-actions[bot]]

:x: Gradle check result for 81bd1e115655274b00750ada5073a3225421b8eb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?