[RW Separation] Introduce allocation filter to control placement of search only replicas

[mch2]

This is a continuation of https://github.com/opensearch-project/OpenSearch/issues/15445 that is still in review, raising this in draft for now.

Description

Introduce allocation filter to control placement of search only replicas.

The new filter cluster.routing.allocation.search.replica.dedicated.include acts as a normal include filter in that you can specify multiple values and search shards will allocate there if one of the attributes is present. However, it is different in that it will exclude any non search shard for all of the attributes given.

Related Issues

Resolves https://github.com/opensearch-project/OpenSearch/issues/15445

Check List

• [ ] Functionality includes testing.
• [ ] API changes companion pull request created, if applicable.
• [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

:x: Gradle check result for 264cc152bd134f8d4c81dc7109f125833e47ae38: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 264cc152bd134f8d4c81dc7109f125833e47ae38: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 95666655ed24148eeae93ba772fd4da51a456209: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for f8b2a81581fc9306442be889d21123e9e68fb659: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 6187f0a6a8a7c2018e885f5e4ce736458bd526b4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 45e354ff5f392d5a07a49a2f47bdc4b30b249865: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for f0e34d21bb5736327f709f15cb2dba43a914e511: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for a4f3103df3bc0321637c0dce35bc04f444813d11: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 20f1217dfea1877ee7dc4046e86b2488c0d15757: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for ad29e7c7cef0ce29b0f4633684c3ea258141b90c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 642ef591709fbea72f2e63f3c0841d647b4da7af: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for da6215b438754d23c8c43d073445ef6be1ed0d8a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 30b1a34d9e960e85119f1e57f6fe72dc9b6cbaad: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 046348a9fc96e7083e57dbfbd14977c835e95dcb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 02977be573d00446fbc4801dd69c3480518600fd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 02977be573d00446fbc4801dd69c3480518600fd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mch2]

❌ Gradle check result for 02977be: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

https://github.com/opensearch-project/OpenSearch/issues/15584

[github-actions[bot]]

:x: Gradle check result for 02977be573d00446fbc4801dd69c3480518600fd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for b5b39bedc04035a7d8cb19fbb88f4910e609056f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 85c21ab0d07becec59a4061d45aaba4372213c7b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:grey_exclamation: Gradle check result for f29278532cca498bec59dc8530b538b10745b790: UNSTABLE

• TEST FAILURES:

      2 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=snapshot.status/10_basic/Get missing snapshot status succeeds when ignoreUnavailable is true}
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=snapshot.status/10_basic/Get snapshot status}
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=snapshot.status/10_basic/Get snapshot status}
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=snapshot.status/10_basic/Get missing snapshot status throws an exception}
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=snapshot.status/10_basic/Get missing snapshot status succeeds when ignoreUnavailable is true}

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 95.65217% with 1 line in your changes missing coverage. Please review.

Project coverage is 71.98%. Comparing base (e087272) to head (05d8e19). Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
...cation/decider/SearchReplicaAllocationDecider.java	94.73%	0 Missing and 1 partial :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15455      +/-   ##
============================================
- Coverage     72.00%   71.98%   -0.02%     
+ Complexity    64149    64126      -23     
============================================
  Files          5269     5270       +1     
  Lines        299834   299890      +56     
  Branches      43330    43343      +13     
============================================
- Hits         215895   215888       -7     
- Misses        66217    66250      +33     
- Partials      17722    17752      +30     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[mch2]

removed changelog entry here to reduce all the backport / gc pain, will include 1 entry for all the separation prs and link to the design issue.

[github-actions[bot]]

:x: Gradle check result for 05d8e19fe69f63d33375620e1d4fe719f15c0bb9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mch2]

FAILURE

https://github.com/opensearch-project/OpenSearch/issues/15692

[github-actions[bot]]

:x: Gradle check result for 05d8e19fe69f63d33375620e1d4fe719f15c0bb9:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mch2]

❌ Gradle check result for 05d8e19:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

https://github.com/opensearch-project/OpenSearch/issues/14328

[github-actions[bot]]

:white_check_mark: Gradle check result for 05d8e19fe69f63d33375620e1d4fe719f15c0bb9: SUCCESS