[FEATURE] Cluster metrics monitor enhancement follow-up items

[AWSHurneyt]

There were non-blocking action items leftover from the enhancement PR (https://github.com/opensearch-project/alerting/pull/992). The action items include:

1. Adding javadoc comments to better explain the logic in the various helper methods
2. Enhancing the tests by calling the actual cat API to compare that response to the monitor execution response.
3. Resolve the issues that are blocking us from refactoring calls to .get() to .suspendUntil(). Ref
4. Add integration tests for all supported API, including path parameters. Related https://github.com/opensearch-project/alerting-dashboards-plugin/issues/186

[AWSHurneyt]

To reduce code duplication, it would also be worthwhile to create an issue in the core repo to ask that the request and response object creation from the API rest handlers be extracted to helper methods that are accessible outside of the rest handler classes.

[AWSHurneyt]

To reduce code duplication, it would also be worthwhile to create an issue in the core repo to ask that the request and response object creation from the API rest handlers be extracted to helper methods that are accessible outside of the rest handler classes.

May need to investigate whether this is still viable. The _cat/indices, and _cat/shards API are not longer available on large domains. To support the new _list API, cluster metrics monitors would need to be enhanced to support query params (e.g., the next_token, and size params mentioned in the below issues).

• https://github.com/opensearch-project/OpenSearch/issues/14258
• https://github.com/opensearch-project/OpenSearch/issues/14257

Those changes should not impact the current cluster metrics monitors implementation as _cat/indices, and _cat/shards monitors do not call those API directly; they instead mimic the sequence of calls made by those API. https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/kotlin/org/opensearch/alerting/util/clusterMetricsMonitorHelpers/SupportedClusterMetricsSettingsExtensions.kt#L47