Master node changes for master task throttling

[dhwanilpatel]

Master node changes for master task throttling

Signed-off-by: Dhwanil Patel [email protected]

Description

This is one of multiple PR planned for master task throttling. In this PR we are making changes in MasterTaskThrottler and MasterService.

MasterTaskThrottler: It contains the settings of throttling and extends Throttler class for the use case of task throttling. MasterService: Plugged the MasterTaskThrottler on submitting new tasks.

We have introduced the throttling settings as well, using which we can set the throttling limits for different tasks. By default throttling will be disabled and will be enabled on those task for which it is being configured via dynamic setting. Below is sample on how settings can be updated for throttling.

curl -XPUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' 
{ 
  "persistent": 
    { 
      "master.throttling.thresholds" : 
      { 
        "put-mapping" : 
        { 
          "value" : 10 
        }
      } 
    } 
}'

Issues Resolved

Relates to : https://github.com/opensearch-project/OpenSearch/issues/479

Check List

• [x] New functionality includes testing.
  • [x] All tests pass
• [ ] New functionality has been documented.
  • [ ] New functionality has javadoc added
• [x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/519/
• CommitID: fb05a699af44faaf6fe4183432c16a9b58f35267

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/793/
• CommitID: 3ee75efdd44fada9c7d3ce19815eaee8c9817f69

[dhwanilpatel]

@Bukhtawar / @shwetathareja / @gbbafna I have made changes for maintaining ConcurrentHashMap for maintaining count per task type and making throttling decision based on count. I have created below Listener and MasterTaskThrottler will be implementing it and update the count per task type.

    void onSubmit(List<? extends TaskBatcher.BatchedTask> tasks);
    void onProcessed(List<? extends TaskBatcher.BatchedTask> tasks);
    void onSubmitFailure(List<? extends TaskBatcher.BatchedTask> tasks);

We can keep this Listener at the individual task level into BatchedTask, but I dont see any usecase where we would want to perform various activity whenever individual task gets submit/timeout/execute. If anyone has any use case in mind, please share it.

To keep it simple now, I have kept the Listener at the TaskBatcher level, task batcher will call relevant methods based on activity it performs on tasks.

I have made the changes for it, and updating the PR to get the early feedback on approach. This will break few of the UTs, I will fix the UTs in upcoming revisions.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/1618/
• CommitID: 6c99e1bdf008bef15af9b6e3906cf0a628be28ee

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/1695/
• CommitID: 2fad544a730a04c5120ee0358a041bd3eb33bb0f

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/1827/
• CommitID: bfdd946e1c7d65e46e1d69f367141366e7bd6607

[dhwanilpatel]

start gradle check

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/2062/
• CommitID: bfdd946e1c7d65e46e1d69f367141366e7bd6607

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/2063/
• CommitID: 41bec655697e707851ecb6912be9fbd14c8fd030

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/2137/
• CommitID: e408f67abe465256fbd74a58bbcbc67e5a414132

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS :white_check_mark:
• URL: https://build.ci.opensearch.org/job/gradle-check/2139/
• CommitID: a8604709f86bd6c45897a9d78a2e512cfcf7da7d

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS :white_check_mark:
• URL: https://build.ci.opensearch.org/job/gradle-check/2245/
• CommitID: eaefc71524f4183b0d6ec74eeb62137e3b114f60

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/2515/
• CommitID: 53b661488dc25262a1e447d46cdef0ed37740439

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/2520/
• CommitID: 018912be723e1ab313cd44245becfa8dc297496b

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS :white_check_mark:
• URL: https://build.ci.opensearch.org/job/gradle-check/2527/
• CommitID: 3c8b6d78716dfd94d60631663ccd0eff17bd8a56