Added in-flight cancellation of SearchShardTask based on resource consumption

[ketanv3]

Description

This feature aims to identify and cancel resource intensive SearchShardTasks if they have breached certain thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the cluster performance.

Issues Resolved

#1181

Check List

• [x] New functionality includes testing.
  • [ ] All tests pass
• [x] New functionality has been documented.
  • [x] New functionality has javadoc added
• [ ] Commits are signed per the DCO using --signoff
• [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ketan Verma [email protected]

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/3294/
• CommitID: f613ae571c1371cfe5c492e74e7d3acf7f6e8143

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/3297/
• CommitID: ad0454e14298dd97462e991a661ffca742f12292

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/3296/
• CommitID: 7a7a952f1cd84c7db15bcc289593bfcb58e9a445

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/3321/
• CommitID: 7786c5eed1fb7004d1ce55560f1436ae7aa08c35

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS :white_check_mark:
• URL: https://build.ci.opensearch.org/job/gradle-check/3327/
• CommitID: 4a9abafb694d7bbd7128c153905c558629ea9518

[codecov-commenter]

Codecov Report

Merging #4565 (4a9abaf) into main (2c27dfd) will increase coverage by 0.12%. The diff coverage is 77.14%.

@@             Coverage Diff              @@
##               main    #4565      +/-   ##
============================================
+ Coverage     70.51%   70.64%   +0.12%     
- Complexity    57310    57515     +205     
============================================
  Files          4628     4642      +14     
  Lines        276073   276579     +506     
  Branches      40421    40466      +45     
============================================
+ Hits         194677   195378     +701     
+ Misses        65103    64825     -278     
- Partials      16293    16376      +83     

Impacted Files	Coverage Δ
...min/cluster/stats/TransportClusterStatsAction.java	70.83% <ø> (ø)
...ain/java/org/opensearch/cluster/ClusterModule.java	100.00% <ø> (ø)
...rg/opensearch/common/settings/ClusterSettings.java	91.89% <ø> (ø)
...ain/java/org/opensearch/tasks/CancellableTask.java	82.35% <ø> (-5.89%)	:arrow_down:
...search/cluster/MockInternalClusterInfoService.java	0.00% <0.00%> (ø)
.../java/org/opensearch/test/InternalTestCluster.java	60.72% <ø> (-1.50%)	:arrow_down:
...rch/action/admin/cluster/node/stats/NodeStats.java	50.67% <33.33%> (-1.13%)	:arrow_down:
...src/main/java/org/opensearch/node/NodeService.java	72.41% <33.33%> (-0.81%)	:arrow_down:
...arch/backpressure/trackers/ElapsedTimeTracker.java	61.29% <61.29%> (ø)
.../search/backpressure/trackers/CpuUsageTracker.java	62.06% <62.06%> (ø)
... and 502 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.