Initial commit to support a search only replica for RW separation.

[mch2]

Description

Initial commit for the search only replica for reader writer split. This PR contains the following:

1. Introduce searchOnly flag on ShardRouting.
2. Add feature flag to enable/disable the feature.
3. support both create and update APIs to toggle search replica count.
4. Changes to exclude search replicas from primary eligibility.
5. Changes to prevent any replicationOperation from routing to a search replica.
6. Small change to return [s] instead of [r] for search replicas in /_cat/shards

Related Issues

Resolves #15368 related https://github.com/opensearch-project/OpenSearch/issues/15306

Check List

• [x] Functionality includes testing.
• [ ] API changes companion pull request created, if applicable.
• [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

:x: Gradle check result for a1b71a9c151992c505e144466a7680ed3959f12e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for b4c8d03810c178b7500a1249b82281d138a8893f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for e50f2dc762022ab476f6600eb0078f0d5ddf5eed: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for ff33b285b0d5c7410e6e6e3bd658a8458c00f2d8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for e448e1fed601b4facd0ba1b99b572abe5c550c7f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mch2]

❌ Gradle check result for e448e1f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

https://github.com/opensearch-project/OpenSearch/issues/15427

[github-actions[bot]]

:white_check_mark: Gradle check result for e448e1fed601b4facd0ba1b99b572abe5c550c7f: SUCCESS

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 78.87324% with 30 lines in your changes missing coverage. Please review.

Project coverage is 71.88%. Comparing base (1d5082e) to head (2207a10). Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
.../opensearch/cluster/routing/IndexRoutingTable.java	84.61%	2 Missing and 4 partials :warning:
...a/org/opensearch/cluster/routing/ShardRouting.java	71.42%	3 Missing and 3 partials :warning:
...a/org/opensearch/cluster/routing/RoutingTable.java	76.19%	2 Missing and 3 partials :warning:
...a/org/opensearch/cluster/routing/RoutingNodes.java	0.00%	0 Missing and 4 partials :warning:
...java/org/opensearch/cluster/metadata/Metadata.java	66.66%	1 Missing and 1 partial :warning:
...h/cluster/metadata/MetadataCreateIndexService.java	77.77%	0 Missing and 2 partials :warning:
...luster/metadata/MetadataUpdateSettingsService.java	91.30%	0 Missing and 2 partials :warning:
...g/opensearch/rest/action/cat/RestShardsAction.java	33.33%	1 Missing and 1 partial :warning:
...s/replication/SegmentReplicationTargetService.java	0.00%	0 Missing and 1 partial :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15410      +/-   ##
============================================
- Coverage     72.01%   71.88%   -0.14%     
+ Complexity    63603    63575      -28     
============================================
  Files          5247     5247              
  Lines        297186   297308     +122     
  Branches      42939    42977      +38     
============================================
- Hits         214023   213714     -309     
- Misses        65610    66007     +397     
- Partials      17553    17587      +34     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[andrross]

@mch2 How does allocation work here? Is the search node role being used? I'm assuming the most common use case here will be to allocate the search replicas to a distinct set of nodes in order to get physical isolation.

[mch2]

@mch2 How does allocation work here? Is the search node role being used? I'm assuming the most common use case here will be to allocate the search replicas to a distinct set of nodes in order to get physical isolation.

I've made no changes to allocation here to split up the PRs. With this one the search shards would be allocated as normal replicas and would live next to regular replicas. With the next PR I'll be introducing a decider that can physically isolate based on node attributes instead of forcing use of the search node role - more on why is here in the Node Separation section.

[mch2]

adding more units here for coverage...

[github-actions[bot]]

:x: Gradle check result for 2caa87f86670c434d13b475cbb33ef3b8bbb7dbb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 9c54a4e41bc8dac339ebeb54643c52a32a6f7f35: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for b1938d8f861dc6abd70854b4f82a8ba6e43b05c8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for c059bd8aba78de2822e541582a9de647c1feacef: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mch2]

❌ Gradle check result for c059bd8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

https://github.com/opensearch-project/OpenSearch/issues/14289

[github-actions[bot]]

:white_check_mark: Gradle check result for 6e563e50c7fb74bae29af4d4a1e5a59c9f722ed6: SUCCESS

[github-actions[bot]]

:x: Gradle check result for ea244584c71d08f264024d4bdbb2c446a58cd57b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for ea244584c71d08f264024d4bdbb2c446a58cd57b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mch2]

❌ Gradle check result for ea24458: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

https://github.com/opensearch-project/OpenSearch/issues/15117

[mch2]

❌ Gradle check result for ea24458: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

https://github.com/opensearch-project/OpenSearch/issues/12344

[github-actions[bot]]

:grey_exclamation: Gradle check result for a42af7e1288192c622caaa42b77d59f49bcf3ee7: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions[bot]]

:x: Gradle check result for c5d29a9f60b536a10a585641e1a2632be1931288: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for e182c9ea9f7d36a0874f0b7ea2f12a2734d2ff97: SUCCESS

[mch2]

❌ Gradle check result for c5d29a9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

https://github.com/opensearch-project/OpenSearch/issues/10820

[github-actions[bot]]

:x: Gradle check result for 612c2c34039b44225696e0e5236b7cf87e7f3e7b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for 2207a10f6ca42ac0587965155a9c6acf63998d74: SUCCESS

[opensearch-trigger-bot[bot]]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15410-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1e9fdb452101476a24737cc1b3aa1fad15df8fca
# Push it to GitHub
git push --set-upstream origin backport/backport-15410-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15410-to-2.x.