Star Tree File Formats

[sarthakaggarwal97]

Description

Introducing the file formats for composite index. There could be multiple implementations of composite index, star-tree being one of them.

With this change, we introduce four new files.

1. cid - Composite Index Data
2. cim - Composite Index Meta
3. cidvd - Composite Doc Values Data
4. cidvm - Composite Doc Values Meta

Related Issues

Resolves #14837

Check List

• [ ] Functionality includes testing.
• [ ] API changes companion pull request created, if applicable.
• [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

:x: Gradle check result for b93a0ce4351b425a0357dfb410e330f12ae8b24e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 94d1928030798d71745d9b34ed7a1d9e7c641798: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for 26337a87fe364c3970292fcf966cea5a95da0ffb: SUCCESS

[github-actions[bot]]

:white_check_mark: Gradle check result for 1b5009e7ea004898470d23d09d69e3726ba6caba: SUCCESS

[github-actions[bot]]

:x: Gradle check result for 1fc9add31832725bcf3c2daf9d3edb8ef72efa9c:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for 72ef06fd8a64c1ab4dda145f5aac149f7c3606dd: SUCCESS

[github-actions[bot]]

:x: Gradle check result for 9250579d2b7559788ad712fa683145ee1bea55d8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for cb32164a12212e0d5ca736d6945e1619a70e09bc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mgodwan]

Can this PR be broken down? One way I see is to use separate PRs for format write, format read, and integration with build flow. Otherwise it may take time to review this in a single go.

[github-actions[bot]]

:white_check_mark: Gradle check result for 7ded5caadb34fb01a2df45da6da233ba22b2fadf: SUCCESS

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 89.72746% with 49 lines in your changes missing coverage. Please review.

Project coverage is 72.07%. Comparing base (758c2aa) to head (e13e400). Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...posite/composite99/Composite99DocValuesReader.java	77.10%	11 Missing and 8 partials :warning:
...eindex/datacube/startree/index/StarTreeValues.java	81.53%	7 Missing and 5 partials :warning:
...datacube/startree/builder/BaseStarTreeBuilder.java	96.12%	3 Missing and 3 partials :warning:
...index/datacube/startree/node/InMemoryTreeNode.java	88.46%	0 Missing and 6 partials :warning:
...posite/composite99/Composite99DocValuesWriter.java	93.47%	1 Missing and 2 partials :warning:
...tacube/startree/builder/OnHeapStarTreeBuilder.java	92.30%	1 Missing :warning:
.../startree/fileformats/data/StarTreeDataWriter.java	93.33%	0 Missing and 1 partial :warning:
...rtree/fileformats/meta/StarTreeMetadataWriter.java	83.33%	1 Missing :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14809      +/-   ##
============================================
+ Coverage     72.02%   72.07%   +0.04%     
- Complexity    63769    63904     +135     
============================================
  Files          5249     5249              
  Lines        297795   298087     +292     
  Branches      43034    43071      +37     
============================================
+ Hits         214480   214838     +358     
+ Misses        65735    65633     -102     
- Partials      17580    17616      +36     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[github-actions[bot]]

:white_check_mark: Gradle check result for 8946d856de962f0f87d3ba65bafed1bd1b615d28: SUCCESS

[github-actions[bot]]

:x: Gradle check result for 40015c47d34cd87da757f54fb60eae325c0dd1e0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 19e4a958e8fbe277dad1a90b38804c4f5f61cc64: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for fe631b4ee47df9532e58e5ff6acce856ed490f30: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 969995af81ee0fd996ae0968cea17688b51ab8b5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 63d0446bd2f1fcdba97fa8fd97380f8639b22b18: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[bharath-techie]

Lets ensure

• Writer and reader are mirrors of each other
• We only initialize meta and indexinput during reader initialization
• If a new composite type gets introduced, we should easily be able to fit in without changing existing classes as much as possible

[github-actions[bot]]

:white_check_mark: Gradle check result for fa1c948bcad1e85cce4564b2ee48a4de83356bce: SUCCESS

[github-actions[bot]]

:x: Gradle check result for 05b64f146d1784f2e73a41ac711a46980c6424f8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:grey_exclamation: Gradle check result for 1521ed166119b0ce2880b2835a82448b4c36ae22: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions[bot]]

:x: Gradle check result for 48c0623484a1deb53130f0347022ed7f29163e29: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:grey_exclamation: Gradle check result for 39505cdb65b032d683607a46c23448420ba75d0f: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions[bot]]

:x: Gradle check result for 18c5f1471a1b212773f099e5b2022919c300deb5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 1a16dbfb55b05d3601cba5e9bc29ca4706eb60cb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for c96274a19699b46d910ae911b398c81709d2c482: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 4831683d573a40d64c7e07eb5acb00cd5fc74b5a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 6d9052ce4c7af0f6beb6205fee50bd0271426edd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 1715a4c170960ca689e1201a542bee9c8e029b64: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 3413fa4e23f22bd7360cf4f5c233fd1524c41f73: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?