[BUG] Problem with getting ctx.results

[zil1]

When sending an alert, ctx.result is returned {}, while ctx.error does not return anything. When trying to send ctx.result.0 , the message is not sent, and an error appears in the logs:

[2022-06-29T11:48:44,800][ERROR][o.o.s.m.MustacheScriptEngine] [opensearch-node03] Error running com.github.mustachejava.codes.DefaultMustache@6422caa7
com.github.mustachejava.MustacheException: Failed to get value for ctx.results.0.hits.total.value @[query-template:14]
	at com.github.mustachejava.codes.ValueCode.execute(ValueCode.java:74) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.codes.DefaultMustache.run(DefaultMustache.java:34) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.codes.DefaultCode.execute(DefaultCode.java:162) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.codes.DefaultMustache.execute(DefaultMustache.java:57) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.Mustache.execute(Mustache.java:38) ~[compiler-0.9.10.jar:?]
	at org.opensearch.script.mustache.MustacheScriptEngine$MustacheExecutableScript.lambda$execute$0(MustacheScriptEngine.java:137) ~[lang-mustache-client-2.0.1.jar:2.0.1]
	at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
	at org.opensearch.script.mustache.MustacheScriptEngine$MustacheExecutableScript.execute(MustacheScriptEngine.java:136) [lang-mustache-client-2.0.1.jar:2.0.1]
	at org.opensearch.alerting.MonitorRunnerService.compileTemplate$opensearch_alerting(MonitorRunnerService.kt:271) [opensearch-alerting-2.0.1.0.jar:2.0.1.0]
	at org.opensearch.alerting.MonitorRunner.runAction(MonitorRunner.kt:62) [opensearch-alerting-2.0.1.0.jar:2.0.1.0]
	at org.opensearch.alerting.DocumentLevelMonitorRunner.runForEachDocTrigger(DocumentLevelMonitorRunner.kt:276) [opensearch-alerting-2.0.1.0.jar:2.0.1.0]
	at org.opensearch.alerting.DocumentLevelMonitorRunner.runMonitor(DocumentLevelMonitorRunner.kt:194) [opensearch-alerting-2.0.1.0.jar:2.0.1.0]
	at org.opensearch.alerting.DocumentLevelMonitorRunner$runMonitor$1.invokeSuspend(DocumentLevelMonitorRunner.kt) [opensearch-alerting-2.0.1.0.jar:2.0.1.0]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]
Caused by: com.github.mustachejava.MustacheException: 0 @[query-template:14]
	at com.github.mustachejava.codes.DefaultCode.get(DefaultCode.java:150) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.codes.ValueCode.execute(ValueCode.java:62) ~[compiler-0.9.10.jar:?]
	... 17 more
Caused by: java.lang.IndexOutOfBoundsException: 0
	at org.opensearch.common.util.iterable.Iterables.get(Iterables.java:114) ~[opensearch-2.0.1.jar:2.0.1]
	at org.opensearch.script.mustache.CustomReflectionObjectHandler$CollectionMap.get(CustomReflectionObjectHandler.java:143) ~[lang-mustache-client-2.0.1.jar:2.0.1]
	at org.opensearch.script.mustache.CustomReflectionObjectHandler$CollectionMap.containsKey(CustomReflectionObjectHandler.java:152) ~[lang-mustache-client-2.0.1.jar:2.0.1]
	at com.github.mustachejava.reflect.ReflectionObjectHandler.findWrapper(ReflectionObjectHandler.java:146) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.reflect.ReflectionObjectHandler.find(ReflectionObjectHandler.java:87) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.reflect.GuardedBinding.getWrapper(GuardedBinding.java:103) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.reflect.GuardedBinding.createAndGet(GuardedBinding.java:89) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.reflect.GuardedBinding.get(GuardedBinding.java:84) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.codes.DefaultCode.get(DefaultCode.java:145) ~[compiler-0.9.10.jar:?]
	at com.github.mustachejava.codes.ValueCode.execute(ValueCode.java:62) ~[compiler-0.9.10.jar:?]
	... 17 more

How can one reproduce the bug? Steps to reproduce the behavior:

1. Create a monitor with the type "Per document monitor"
2. In the message specify a variable {{ctx.results.0.hits.total.value}}

What is your host/environment?

• Docker
• OpenSearch v2.0.1
• OpenSearch Dashboards v2.0.1

[rishabhmaurya]

@zil1 could you add what the query and output looks like?

[zil1]

Slack destination: Message:

- Period start: {{ctx.periodStart}}
- Period end: {{ctx.periodEnd}}
results: {{ctx.results}}
error: {{ctx.error}}

Message in Slack:

Alerting-Notification Action



- Period start: 2022-07-18T07:34:48.345315097Z
- Period end: 2022-07-18T07:34:48.345315685Z
results: {}
error:

[eirsep]

@zil1 I see that you have raised a concern for "Per document monitor" in specific.

For per doc monitors, have you checked out Document Findings?

In ctx, we have the alerts field i.e. ctx.alerts which would be the core information informing us which document id caused the alert and the id of document in findings index.

Here is the sample context body from a dummy alert I created in a Per document monitor for reference:

results: {monitor={_id=xNPqIoIByUf5KUWbzBd4, _version=5, name=test, enabled=true}, results=[], periodStart=2022-07-21T23:33:49.369701176Z, periodEnd=2022-07-21T23:33:49.369701760Z, error=null, trigger={id=w9PqIoIByUf5KUWbzBcr, name=Trigger1, severity=1, actions=[{name=testChimeAlert479Issue}]}, alerts=[{acknowledged_time=null, id=3a245826-bb0c-4a0e-86b9-fbc9526b9329, version=-1, end_time=null, error_message=null, last_notification_time=1658446429380, severity=1, start_time=1658446429380, state=ACTIVE, bucket_keys=null, parent_bucket_path=null, finding_ids=9cb82832-b0ea-4f75-b756-b88086afd2a2, related_doc_ids=r8AbI4IB8lJ1LO2BiMZa|test}]}

A monitor can run multiple queries in execution but ctx.alerts would contain all the information relevant to that particular document and alert.

Do you have any use case requiring you to consume results of all the queries run by the monitor in that execution?

[zil1]

@eirsep, for example, I want to get the "message" field in a document

message: {{ ctx.results.0.hits.hits.0._source.message }}

[getsaurabh02]

@zil1 Just to confirm on the ask here, are you suggesting that in addition to the document findings which are part of the alerts today (which contains the specific document id which caused the alerts), you also would want to have the same set of document ids part of the results field as well?

[zil1]

@getsaurabh02 yes, if possible

[getsaurabh02]

/cc: @brijos to provide some context on the behavior.

[df3rry]

Not sure if its the same use-case; but I personally would like to send document context (actual fields of a document) in a per-document monitor. For example, you have syslog -> logstash -> opensearch and you want to monitor logs from different hosts that have a certain severity/message ie. severity:crit and message:"something superbad" ideally the actions should be able to notify the channel with the context "hostX just logged a crit alert - something superbad". Apologies if this is documented somewhere, I really have been unable to find the best way of achieving this.

[waza-ari]

I'd like to second this request. It allows the creation of much more useful alert messages, which contain key information for troubleshooting without first looking up. For me, the document includes a correlation_id, which would be incredibly useful to show directly in the alert manage.

As its a per document monitor, it would be very useful to access the document that triggers the alert.

[cpholt]

I'll give another up-vote on this. Having a per-document alert that includes the content of the document in the alert is the one thing I haven't been able to find a way to do. I found this discussion after my trials and it seems I'm not the first person with this request.

It looks like the change was done, but not merged because there wasn't an easy way to test? Has there been any more movement on that front, or has this request fallen off the priority list?

[ghobadimhd]

I also need this.

[nik06]

Facing similar issue and seems like this feature is much needed. Accessing the document fields directly using ctx rather than reference would ease in generating an Action message that is clear in first place. @lezzago

[engechas]

Sample document support was added in #1450. Can you check if that solves the issues you are facing?