[RFC] Query Insights Dashboard

[ansjcy]

Is your feature request related to a problem? Please describe. With the Macro and Micro visibility metrics collected and aggregated by the query insight framework (https://github.com/opensearch-project/OpenSearch/issues/11429), we can create a dedicated dashboard to visualize those metrics to provide an end to end solution to OS users to enable them to view point in time (e.g. last one hour) query insights.

Describe the solution you'd like The dashboard should have multiple views in terms of different query insight data, and for each type of data, it should provide an overview and drill down view for the search queries. Overview shows the trend (top queries with latency, query categorization) with time, and the drill down view can provide a flow graph to show detailed data (like latency and resource distribution) across different stages of the query.

For example, for the top-n queries feature (https://github.com/opensearch-project/OpenSearch/issues/11186), the query insight dashboard for "overview" should be similar to the below mock:

And if you drill down into this query, you should see the query details and the resource consumption in those important phases/stages/workflows.

Describe alternatives you've considered N/A

Additional context Some interesting discussions around this topic in the comments of: https://github.com/opensearch-project/OpenSearch/issues/11186