[FEATURE] Alerting module alerts without data that triggers the alert present

[sandervandegeijn]

Describe the bug

Noticed the same behavior for multiple alerts we are using. Alerts goes off, but can't find the data that triggered it anywhere. When we were using open distro we did not encounter these problems, although I now seperated the master / data nodes (3 master, 3 data. With open distro we were using 3 nodes that did everything).

Definiton and graph showing 0 results

Yet, history shows red:

Data:

no data present (look at index names, nothing matching the prd indexes)

To Reproduce See above

Expected behavior No alerts when there is no data present that matches the rules

OpenSearch Version 1.0.0

Dashboards Version 1.0.0

Plugins

Default docker image

Screenshots

Host/Environment (please complete the following information): Kubernetes

Additional context

Add any other context about the problem here.

[kavilla]

Hello @ict-one-nl, same as the other issue.

I believe this would be an enhancement for the alerting plugin .

@opensearch-project/admin could we get this issue transferred to alerting plugin repo.

Thanks!