Make thread pool sizes configurable

[dbwiddis]

Description

Allows configuration of thread pool sizes at startup using a static setting.

Check List

• [x] New functionality includes testing.
• [x] New functionality has been documented.
• ~[ ] API changes companion pull request created.~
• [x] Commits are signed per the DCO using --signoff.
• [X] Public documentation issue/PR created.
  • https://github.com/opensearch-project/documentation-website/pull/9881

Reviewers please provide tech review of above doc PR.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 77.37%. Comparing base (7aec38e) to head (c2e71a3). Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1139      +/-   ##
============================================
+ Coverage     77.33%   77.37%   +0.04%     
- Complexity     1096     1099       +3     
============================================
  Files           101      101              
  Lines          5308     5318      +10     
  Branches        511      511              
============================================
+ Hits           4105     4115      +10     
  Misses          954      954              
  Partials        249      249              

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[junweid62]

LGTM! Just curious — why do we need this feature?

[dbwiddis]

LGTM! Just curious — why do we need this feature?

The existing sizes are assuming a single tenant cluster. For Multi-tenant implementation with many more tenants, we need to be able to have more simultaneous tenant provisioning.

I honestly think our thread pool sizes are too small as we're mostly just blocking waiting on the result of async REST calls.

[dbwiddis]

Tests failing due to https://github.com/opensearch-project/job-scheduler/pull/773, need upstream dependencies who depend on Guava to update.