Add rest, transport, service layer changes for Tiering

[neetikasinghal]

Description

Adding rest, transport, service layer changes for hot to warm tiering

Related Issues

https://github.com/opensearch-project/OpenSearch/issues/14545

Pending - Integration with composite directory, shard recovery flow.

Check List

• [ ] New functionality includes testing.
  • [ ] All tests pass
• [ ] New functionality has been documented.
  • [ ] New functionality has javadoc added
• [ ] API changes companion pull request created.
• [ ] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• [ ] Commits are signed per the DCO using --signoff
• [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• [ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

:x: Gradle check result for 4cb595266d413fe1b18e089c36ba8cc6df7c99fd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 2a315ef5bc48ec37f0ddfcf9b60f9b98cce13fd9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for b3859b7c6ffd09a3b4d018c235b513f7036b7a91: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 8099f623e281ce6fd30d735b126aafdf3e9836f5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for c3877a27a9f9bf92e0006a968a0f3f4945357f3c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for dc04eab338294efaf069ebc9f9aa90d1db839f7d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 2f14593372cbb6d0bc7f077d759cb2d600d6b689: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[dblock]

@andrross There's an API change here that's unusual, /{index}/_tier/warn|cold|etc, wdyt?

[github-actions[bot]]

:x: Gradle check result for 7810a6ddb19b920eafba19012477b0dff3694bfb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 730d14854963954f94101e743604a1ee0ed9b125: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 5569b43b65eef2b2b512ed9f43044489b9089985: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for da7c5c2638dc44f7c0089f0dba2abf4219b78b4c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 66ceaa130e9e9af7c705e266956fadbf540ce4e5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 66ceaa130e9e9af7c705e266956fadbf540ce4e5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 9f55e7820c7ad4fe6b149803cc80fa6a26f44634: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for b1107eca0318d8ae726c3519bccafc2a1e51a899: SUCCESS

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 74.07407% with 70 lines in your changes missing coverage. Please review.

Project coverage is 71.75%. Comparing base (97f26cc) to head (31256f3). Report is 1 commits behind head on main.

Files	Patch %	Lines
...ion/admin/indices/tiering/TieringIndexRequest.java	73.13%	10 Missing and 8 partials :warning:
...earch/indices/tiering/TieringRequestValidator.java	78.04%	12 Missing and 6 partials :warning:
...dmin/indices/tiering/HotToWarmTieringResponse.java	79.16%	9 Missing and 1 partial :warning:
...n/admin/indices/tiering/RestWarmTieringAction.java	23.07%	10 Missing :warning:
...admin/indices/tiering/TieringValidationResult.java	58.33%	9 Missing and 1 partial :warning:
...dices/tiering/TransportHotToWarmTieringAction.java	80.00%	2 Missing and 2 partials :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13980      +/-   ##
============================================
- Coverage     71.86%   71.75%   -0.12%     
  Complexity    62539    62539              
============================================
  Files          5149     5156       +7     
  Lines        293450   293719     +269     
  Branches      42428    42470      +42     
============================================
- Hits         210899   210750     -149     
- Misses        65120    65621     +501     
+ Partials      17431    17348      -83     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[github-actions[bot]]

:x: Gradle check result for 31256f3c584fbd21c99fe589cb485749c83ad775: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[neetikasinghal]

❌ Gradle check result for 31256f3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

failed test: org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting this is flaky: https://github.com/opensearch-project/OpenSearch/issues/14331

[github-actions[bot]]

:grey_exclamation: Gradle check result for 31256f3c584fbd21c99fe589cb485749c83ad775: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[neetikasinghal]

❕ Gradle check result for 31256f3: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

flaky tests https://github.com/opensearch-project/OpenSearch/issues/14318 https://github.com/opensearch-project/OpenSearch/issues/14327

[opensearch-trigger-bot[bot]]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-13980-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c82a282351b4e913d57b50d2fef94d4f046b155c
# Push it to GitHub
git push --set-upstream origin backport/backport-13980-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-13980-to-2.x.

[opensearch-trigger-bot[bot]]

The backport to 2.16 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.16 2.16
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.16
# Create a new branch
git switch --create backport/backport-13980-to-2.16
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c82a282351b4e913d57b50d2fef94d4f046b155c
# Push it to GitHub
git push --set-upstream origin backport/backport-13980-to-2.16
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.16

Then, create a pull request where the base branch is 2.16 and the compare/head branch is backport/backport-13980-to-2.16.

[jed326]

@neetikasinghal could you please open manual backport PRs for both the 2.x and 2.16 branches?

[dblock]

This will also need API specs in https://github.com/opensearch-project/opensearch-api-specification please.