[FEATURE] Dynamic control for metrics collectors

[ansjcy]

Is your feature request related to a problem? The primary objective of this request is to empower administrators and users with dynamic control over metrics collectors during runtime within the OpenSearch ecosystem.

What solution would you like? Specifically, we aim to implement the ability to enable or disable collectors on-the-fly. This dynamic control should intelligently handle cases where a collector's deactivation impacts other dependent collectors, ensuring a seamless and coherent metrics collection process. Furthermore, we aspire to explore the realm of automation by devising mechanisms that automatically adjust collector states based on real-time resource consumption and error metrics. By doing so, we will not only enhance the flexibility and robustness of the metrics collection process but also contribute to the efficiency of the OpenSearch cluster management.

What alternatives have you considered? N/A

Do you have any additional context? A list of collectors we have:

• https://github.com/opensearch-project/performance-analyzer/tree/main/src/main/java/org/opensearch/performanceanalyzer/collectors
• https://github.com/opensearch-project/performance-analyzer-commons/tree/main/src/main/java/org/opensearch/performanceanalyzer/commons/collectors

[kartva]

I'd love to tackle this issue.

[dblock]

[Triage -- attendees 1, 2, 3, 4, 5, 6, 7]

[dblock]

@DesmondWillowbrook do you want to try and finish the draft PR?