Query insights exporters implementation

[ansjcy]

Description

This PR implements generic query insights exporters to export insights data into local index and debug log. Also added the export functionality for top n queries service with related configuration endpoints.

Related Issues

Resolves #11296

Check List

• [x] New functionality includes testing.
  • [x] All tests pass
• [x] New functionality has been documented.
  • [x] New functionality has javadoc added
• [x] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• [x] Commits are signed per the DCO using --signoff
• [x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• [x] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

Compatibility status:

Checks if related components are compatible with change b676849

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git]

[github-actions[bot]]

:x: Gradle check result for 826542015e783ddc093140cee72dc47b2813c4b2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[deshsidd]

Changelog needs update

[deshsidd]

Looks like we are encountering multiple IT failures : https://build.ci.opensearch.org/job/gradle-check/36137/

[ansjcy]

@scrawfor99 Thanks for the comment! That's a very valid point. Previously, when we implemented our API we only allow Admin users to access the API (related PR in security plugin: https://github.com/opensearch-project/security/pull/4006), Since we are exporting the data to a local index, we should probably do similar things here. Do you know if there's a way to set up permission for certain index? Should we add an index_permissions field in here similar to https://github.com/opensearch-project/security/blob/main/config/roles.yml#L410 ?

Alternatively we can set those indices as system indices. But essentially they are logs and I'm hesitated to do that..

[ansjcy]

Related PR to add index permissions for the exported index: https://github.com/opensearch-project/security/pull/4229

[ansjcy]

Documentation issue created: https://github.com/opensearch-project/documentation-website/issues/6945

[github-actions[bot]]

:x: Gradle check result for b676849de554f767cec4122ed9e78db4d45ca33e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 5c50cc4c9e08554163986c81ebcf97d86af33373: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 049dd40cc6a99a8a40bfa54df13e49efb2acb727:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 88c4a295074b0708a2fbaf31014bf269b62dcfba: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for e8d959c15c5bfd2d4de41cfbea715f3e5da7149a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 1b9fc3113a6ae47f72c8254d7f27b6bf5781dc27: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jainankitk]

@jainankitk -- can you please run through your comments and resolve the ones that have been addressed or highlight the ones that still need work? Thanks!

@ansjcy - It seems I don't have permission to resolve the comments, probably only the author does. Can you please go through them and resolve the obvious ones?

[github-actions[bot]]

:x: Gradle check result for 17f5dfc6a67756682874cb1cf6009e47cc85304a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 0c349fa0e21f04ea84ac4865a3d6c846f651da19: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[ansjcy]

The gradle failure is caused by a flaky test: https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aissue+is%3Aopen+testParentFilterFromInlineLeafInnerHitsNestedQuery

[github-actions[bot]]

:white_check_mark: Gradle check result for 7cda19b43c83ec0b4596b1ae78fe29adc1f2b73f: SUCCESS

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 66.91176% with 45 lines in your changes missing coverage. Please review.

Project coverage is 71.60%. Comparing base (b15cb0c) to head (d546d0d). Report is 361 commits behind head on main.

Files	Patch %	Lines
...lugin/insights/core/service/TopQueriesService.java	23.33%	22 Missing and 1 partial :warning:
...ts/core/exporter/QueryInsightsExporterFactory.java	74.41%	8 Missing and 3 partials :warning:
...gin/insights/core/exporter/LocalIndexExporter.java	77.77%	5 Missing and 1 partial :warning:
...search/plugin/insights/core/exporter/SinkType.java	75.00%	3 Missing :warning:
...in/insights/core/service/QueryInsightsService.java	83.33%	2 Missing :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12982      +/-   ##
============================================
+ Coverage     71.42%   71.60%   +0.18%     
- Complexity    59978    61487    +1509     
============================================
  Files          4985     5077      +92     
  Lines        282275   288677    +6402     
  Branches      40946    41785     +839     
============================================
+ Hits         201603   206721    +5118     
- Misses        63999    64921     +922     
- Partials      16673    17035     +362     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[github-actions[bot]]

:x: Gradle check result for 96adf423849ef042d3f584455347ef24451e146d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[ansjcy]

Failed with flaky tests:

Tests with failures:
 - org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting
 - org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction

[github-actions[bot]]

:x: Gradle check result for 96adf423849ef042d3f584455347ef24451e146d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 96adf423849ef042d3f584455347ef24451e146d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 0ab9c0bbf11fa89ca6f3b33bdf226cae5a1ec504: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for e11b3c648433dadc14e55a2e536c229e061c6b53: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for d171db04670733120bf2c8f36b3005dcdb4ca9e7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for d171db04670733120bf2c8f36b3005dcdb4ca9e7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for d171db04670733120bf2c8f36b3005dcdb4ca9e7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for c4512a0b9f1dcc19bdb7090a592458981075dd45: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for c706304ae843475d5aae8aac4827bfe72100a24c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?