[META] Generic Query Insights Framework

[ansjcy]

Is your feature request related to a problem? Please describe. Meta issue to keep track of all the features and tasks in the scope of Query Insights framework. As for now, there are multiple areas we want to invest to improve the query visibility in OpenSearch

• Basic Query Insights plugin with collect, process, and export capabilities.
• Top queries insights by resource usage / latency.
• Metrics improvements, grouped by query categories.
• Query Insights Dashboards with profiling capabilities.
• Recommendations based on query insights to improve search performance.

Describe the solution you'd like related RFC:

• https://github.com/opensearch-project/OpenSearch/issues/11429
• https://github.com/opensearch-project/OpenSearch/issues/11186
• https://github.com/opensearch-project/OpenSearch/issues/12292
• https://github.com/opensearch-project/OpenSearch/issues/11008
• https://github.com/opensearch-project/OpenSearch/issues/11596
• client side insights (UBI): https://github.com/opensearch-project/OpenSearch/issues/12084
• https://github.com/opensearch-project/OpenSearch/issues/13357

Dashboard

• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/5571
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/571 (We can potentially integrate the Query Insights Dashboard with Profiler UX experience.)

Pending Issues

• [x] https://github.com/opensearch-project/OpenSearch/issues/11762
• [x] https://github.com/opensearch-project/OpenSearch/issues/11520
• [x] https://github.com/opensearch-project/OpenSearch/issues/11295
• [x] https://github.com/opensearch-project/OpenSearch/issues/11431
• [x] https://github.com/opensearch-project/documentation-website/issues/6180
• [ ] https://github.com/opensearch-project/OpenSearch/issues/12399
• [ ] https://github.com/opensearch-project/OpenSearch/issues/11296
• [ ] https://github.com/opensearch-project/OpenSearch/issues/12740
• [ ] https://github.com/opensearch-project/documentation-website/issues/6945
• [ ] https://github.com/opensearch-project/OpenSearch/issues/13369

Ongoing Tasks

• [x] https://github.com/opensearch-project/OpenSearch/pull/11526
• [x] https://github.com/opensearch-project/OpenSearch/pull/11903
• [x] https://github.com/opensearch-project/OpenSearch/pull/11904
• [x] https://github.com/opensearch-project/documentation-website/pull/6261
• [x] https://github.com/opensearch-project/security/pull/4006
• [ ] https://github.com/opensearch-project/OpenSearch/pull/12982
• [ ] https://github.com/opensearch-project/security/pull/4229

POC / Experimental PRs

• [ ] https://github.com/opensearch-project/OpenSearch/pull/11506
• [ ] https://github.com/opensearch-project/OpenSearch/pull/12449
• [ ] https://github.com/opensearch-project/OpenSearch/pull/12473
• [ ] https://github.com/opensearch-project/OpenSearch/pull/12529
• [ ] https://github.com/opensearch-project/OpenSearch/pull/13374

Additional context Please add any tickets in the comments in case I'm missing anything.

[peternied]

[Triage - attendees 1 2 3 4] @ansjcy Thanks for filing this issue