opensearch monitor ctx.result not return valid formate data

[dipenpatel235]

Describe the bug I have cleated opensearch cluster and implementing the opensearch alerting monitor. i have used ctx.result to get data but ctx.result not respond with the proper formate data. please look at the data.

{ 0={_shards={total=327, failed=0, successful=327, skipped=0 }, hits={hits=[ {_index= filebeat-2022.04.10, _type=_doc, _source={agent={hostname= linux, name= linux, id=6c003f79-b287-45f0-b023-f060dd8f6fd1, type=filebeat, ephemeral_id=dc182727-426e-4350-a6f3-a3c71ba1dd8c, version=7.10.2 }, log={file={path=/usr/local/zeek/logs/current/smb_files.log }, offset=281484 }, destination={address=101.5.6.55, port=445, ip=101.5.6.55 }, zeek={session_id=CKPXXc3S94nJLNzT37, smb_files={path=\\PMNCUSTG001.cpwm.us\ACCRW, prev_name=acctg\Audit\RSM\FY21\Year End\J. Accounts Payable & Accrued Expenses\J.9 - WC Accrual.xlsx, times={created=2022-04-10T05: 37: 27.516Z, modified=2022-04-10T05: 37: 27.516Z, accessed=2022-04-10T05: 37: 27.516Z, changed=2022-04-10T05: 37: 27.516Z }, size=0, name=acctg\Audit\RSM\FY21\Year End\J. Accounts Payable & Accrued Expenses\2CB7BADF.tmp, action=SMB: :FILE_RENAME } }, source={address=10.162.241.105, port=64194, ip=10.162.241.105 }, fileset={name=smb_files }, network={community_id=1:PoUK9qQSoHRwUmg8OIx16818pTQ=, protocol=smb, transport=tcp }, tags=[zeek.smb_files ], input={type=log }, @timestamp=2022-04-10T05: 36: 46.837Z, file={path=\\\\PMNCUSTG001.cpwm.us\\ACCRW\acctg\\Audit\\RSM\\FY21\\Year End\\J. Accounts Payable & Accrued Expenses\\2CB7BADF.tmp, size=0, created=2022-04-10T05: 37: 27.516Z, name=acctg\Audit\RSM\FY21\Year End\J. Accounts Payable & Accrued Expenses\2CB7BADF.tmp, ctime=2022-04-10T05: 37: 27.516Z, accessed=2022-04-10T05: 37: 27.516Z, mtime=2022-04-10T05: 37: 27.516Z }, ecs={version=1.5.0 }, related={ip=[ 10.162.241.105, 101.5.6.55 ] }, service={type=zeek }, host={hostname= linux, os={kernel=4.15.0-163-generic, codename=bionic, name=Ubuntu, family=debian, version=18.04.5 LTS (Bionic Beaver), platform=ubuntu }, containerized=false, ip=[ 10.162.235.20, fe44: : 25c: 29ff:fefe: 705a, fe44: : 25c: 29ff:fefe: 7065 ], name= linux, id=6c62ac6455e1438abe5b205a2d596d4f, mac=[ 00: 0b: 29:fe: 70: 5c, 00: 0b: 29:fe: 70: 55 ], architecture=x86_64 }, event={ingested=2022-04-10T05: 37: 37.999199771Z, kind=event, created=2022-04-10T05: 36: 49.732Z, module=zeek, action=SMB: :FILE_RENAME, id=CKPXXc3S94nJLNzT37, type=[connection, protocol, change, info ], category=[network, file ], dataset=zeek.smb_files } }, _id=-r36EYABNrKTiK70TGjV, _score=7.2465186 }, {_index= filebeat-2022.04.10, _type=_doc, _source={agent={hostname=abc, name=abc, id=6c003f79-b287-45f0-b023-f060dd8f6fd1, ephemeral_id=ce9faf88-e3df-4645-9125-b2274693f74b, type=filebeat, version=7.10.2 }, log={file={path=/usr/local/zeek/logs/current/smb_files.log }, offset=3326831 }, destination={address=172.25.175.65, port=445, ip=172.25.175.65 }, zeek={session_id=C0SJHO2hWvlK951oN3, smb_files={times={created=2022-04-10T08: 31: 47.967Z, modified=2022-04-10T08: 31: 48.342Z, accessed=2022-04-10T08: 31: 47.967Z, changed=2022-04-10T08: 31: 48.342Z }, size=608, name=2-WebServices-1\ABFiles\00000000-0000-0000-0000-000000000000\00000000-0000-0000-0000-000000000000\C-1e54-1e59.lsabs.tmp, action=SMB: :FILE_OPEN } }, source={address=172.25.199.141, port=49235, ip=172.25.199.141 }, fileset={name=smb_files }, tags=[zeek.smb_files ], network={protocol=smb, community_id=1:qU7m09MDM+gz6CIA7Ir9yub4/Og=, transport=tcp }, input={type=log }, @timestamp=2022-04-10T08: 31: 49.947Z, file={size=608, created=2022-04-10T08: 31: 47.967Z, name=2-WebServices-1\ABFiles\00000000-0000-0000-0000-000000000000\00000000-0000-0000-0000-000000000000\C-1e54-1e59.lsabs.tmp, ctime=2022-04-10T08: 31: 48.342Z, accessed=2022-04-10T08: 31: 47.967Z, mtime=2022-04-10T08: 31: 48.342Z }, ecs={version=1.5.0 }, related={ip=[ 172.25.199.141, 172.25.175.65 ] }, service={type=zeek }, host={hostname=abc, os={kernel=4.15.0-167-generic, codename=bionic, name=Ubuntu, family=debian, version=18.04.5 LTS (Bionic Beaver), platform=ubuntu }, containerized=false, ip=[ 172.25.110.221, fe70: : 11c: 29ff:fe89:ea, fe80: : 30c: 29fe:fe89:f4 ], name=abc, id=6c62ac6455e1438abe5b205a2d596d4f, mac=[ 00: 0c: 30: 88: 00:ba, 00: 0c: 30: 88: 00:f7 ], architecture=x86_64 }, event={ingested=2022-04-10T08: 32: 00.550410110Z, kind=event, created=2022-04-10T08: 31: 51.058Z, module=zeek, action=SMB: :FILE_OPEN, id=C0SJHO2hWvlK951oN3, category=[network, file ], type=[connection, protocol, info ], dataset=zeek.smb_files } }, _id=FVqZEoABNrKTiK708scr, _score=7.2465186 }, {_index= filebeat-2022.04.10, _type=_doc, _source={agent={hostname=abc, name=abc, id=6c003f79-b287-45f0-b023-f060dd8f6fd1, ephemeral_id=ce9faf88-e3df-4645-9125-b2274693f74b, type=filebeat, version=7.10.2 }, log={file={path=/usr/local/zeek/logs/current/smb_files.log }, offset=3328308 }, destination={address=172.25.140.20, port=445, ip=172.25.140.20 }, zeek={session_id=CyzbiZaPQxGK2iXph, smb_files={times={created=2022-04-10T08: 31: 48.373Z, modified=2022-04-10T08: 31: 48.373Z, accessed=2022-04-10T08: 31: 48.373Z, changed=2022-04-10T08: 31: 48.373Z }, size=0, name=2-WebServices-1\ABFiles\00000000-0000-0000-0000-000000000000\00000000-0000-0000-0000-000000000000\D-1e54-1e59.lsabs.tmp, action=SMB: :FILE_OPEN } }, source={address=172.25.199.141, port=49237, ip=172.25.199.141 }, fileset={name=smb_files }, network={protocol=smb, community_id=1:pluY9ySedEpozYV4vwKSMpMgtKI=, transport=tcp }, tags=[zeek.smb_files ], input={type=log }, @timestamp=2022-04-10T08: 31: 49.959Z, file={size=0, created=2022-04-10T08: 31: 48.373Z, name=2-WebServices-1\ABFiles\00000000-0000-0000-0000-000000000000\00000000-0000-0000-0000-000000000000\D-1e54-1e59.lsabs.tmp, ctime=2022-04-10T08: 31: 48.373Z, accessed=2022-04-10T08: 31: 48.373Z, mtime=2022-04-10T08: 31: 48.373Z }, ecs={version=1.5.0 }, related={ip=[ 172.25.199.141, 172.25.140.20 ] }, service={type=zeek }, host={hostname=abc, os={kernel=4.15.0-167-generic, codename=bionic, name=Ubuntu, family=debian, version=18.04.5 LTS (Bionic Beaver), platform=ubuntu }, containerized=false, ip=[ 172.25.110.221, fe60: : 20c: 28fa:fe89:ea, fe80: : 20c: 29ff:fe89:f4 ], name=abc, id=6c62ac6455e1438abe5b205a2d596d4f, mac=[ 00: 0c: 29: 88: 00:ef, 00: 0c: 29: 88: 00:f5 ], architecture=x86_64 }, event={ingested=2022-04-10T08: 32: 00.551134735Z, kind=event, created=2022-04-10T08: 31: 51.058Z, module=zeek, action=SMB: :FILE_OPEN, id=CyzbiZaPQxGK2iXph, category=[network, file ], type=[connection, protocol, info ], dataset=zeek.smb_files } }, _id=GFqZEoABNrKTiK708scr, _score=7.2465186 }, {_index= filebeat-2022.04.10, _type=_doc, _source={agent={hostname= linux, name= linux, id=6c003f79-b287-45f0-b023-f060dd8f6fd1, ephemeral_id=dc182727-426e-4350-a6f3-a3c71ba1dd8c, type=filebeat, version=7.10.2 }, log={file={path=/usr/local/zeek/logs/current/smb_files.log }, offset=628267 }, destination={address=10.162.190.120, port=445, ip=10.162.190.120 }, zeek={session_id=CHx8pX1jZDloU9HIM4, smb_files={path=\\PMNCUSTG003.cpwm.us\buying, times={created=2022-03-31T23: 28: 02.127Z, modified=2022-04-10T20: 17: 22.221Z, accessed=2022-04-10T20: 17: 24.184Z, changed=2022-04-10T20: 18: 24.039Z }, size=2186660, name=P2M\2023 Product to Market\D21\IM\D26733F.tmp, action=SMB: :FILE_DELETE } }, source={address=10.162.248.235, port=51170, ip=10.162.248.235 }, fileset={name=smb_files }, tags=[zeek.smb_files ], network={protocol=smb, community_id=1:IauTjAsxusY8X5yRXsxOrsKoM/U=, transport=tcp }, input={type=log }, @timestamp=2022-04-10T20: 17: 43.141Z, file={path=\\\\PMNCUSTG003.cpwm.us\\buying\P2M\\2023 Product to Market\\D21\\IM\\D26733F.tmp, size=2186660, created=2022-03-31T23: 28: 02.127Z, name=P2M\2023 Product to Market\D21\IM\D26733F.tmp, ctime=2022-04-10T20: 18: 24.039Z, accessed=2022-04-10T20: 17: 24.184Z, mtime=2022-04-10T20: 17: 22.221Z }, ecs={version=1.5.0 }, related={ip=[ 10.162.248.235, 10.162.190.120 ] }, service={type=zeek }, host={hostname= linux, os={kernel=4.15.0-163-generic, codename=bionic, name=Ubuntu, family=debian, version=18.04.5 LTS (Bionic Beaver), platform=ubuntu }, ip=[ 10.162.239.14, fe80: : 20c: 29ef:fffe: 705a, fa80: : 20c: 29ff:fefe: 7064 ], containerized=false, name= linux, id=6c62ac6455e1438abe5b205a2d596d4f, mac=[ 00: 0c: 29:fe: 70: 5a, 00: 0c: 29:fe: 70: 64 ], architecture=x86_64 }, event={ingested=2022-04-10T20: 18: 33.077108851Z, kind=event, created=2022-04-10T20: 17: 44.858Z, module=zeek, action=SMB: :FILE_DELETE, id=CHx8pX1jZDloU9HIM4, category=[network, file ], type=[connection, protocol, deletion, info ], dataset=zeek.smb_files } }, _id=LA8gFYABNrKTiK70zeP_, _score=7.2465186 }, {_index= filebeat-2022.04.10, _type=_doc, _source={agent={hostname= linux, name= linux, id=6c003f79-b287-45f0-b023-f060dd8f6fd1, type=filebeat, ephemeral_id=dc182727-426e-4350-a6f3-a3c71ba1dd8c, version=7.10.2 }, log={file={path=/usr/local/zeek/logs/current/smb_files.log }, offset=617351 }, destination={address=101.5.6.55, port=445, ip=101.5.6.55 }, zeek={session_id=Ca0uOX14g2i7n2hcFh, smb_files={times={created=2022-04-10T20: 18: 17.518Z, modified=2022-04-10T20: 18: 17.172Z, accessed=2022-04-10T20: 18: 17.971Z, changed=2022-04-10T20: 18: 17.203Z }, size=18076, name=Fixture Spec documents to place orders\Fixture Orders Stores\Vendor price Sheets\DC98D4F.tmp, action=SMB: :FILE_OPEN } }, source={address=10.162.248.212, port=62731, ip=10.162.248.212 }, fileset={name=smb_files }, network={community_id=1:irgS3yBEbzautCRpdH2HZ3+yGzU=, protocol=smb, transport=tcp }, tags=[zeek.smb_files ], input={type=log }, @timestamp=2022-04-10T20: 17: 36.684Z, file={size=18076, created=2022-04-10T20: 18: 17.518Z, name=Fixture Spec documents to place orders\Fixture Orders Stores\Vendor price Sheets\DC98D4F.tmp, ctime=2022-04-10T20: 18: 17.203Z, accessed=2022-04-10T20: 18: 17.971Z, mtime=2022-04-10T20: 18: 17.172Z }, ecs={version=1.5.0 }, related={ip=[ 10.162.248.212, 101.5.6.55 ] }, service={type=zeek }, host={hostname= linux, os={kernel=4.15.0-163-generic, codename=bionic, name=Ubuntu, family=debian, version=18.04.5 LTS (Bionic Beaver), platform=ubuntu }, containerized=false, ip=[ 10.162.239.14, fe89: : 20c: 20ff:fefe: 710a, fe80: : 50c: 29ff:fefe: 7064 ], name= linux, id=6c62ac6455e1438abe5b205a2d596d4f, mac=[ 00: 0c: 29:fe: 70: 5a, 00: 0c: 29:fe: 70: 64 ], architecture=x86_64 }, event={ingested=2022-04-10T20: 18: 26.405229445Z, kind=event, created=2022-04-10T20: 17: 37.597Z, module=zeek, action=SMB: :FILE_OPEN, id=Ca0uOX14g2i7n2hcFh, category=[network, file ], type=[connection, protocol, info ], dataset=zeek.smb_files } }, _id=hg8gFYABNrKTiK70s8Lq, _score=7.2465186 }, ], total={value=10000, relation=gte }, max_score=7.2465186 }, took=13209, timed_out=false }

Please look at the screen short this is not look like json or dictionary format:

Expected Result

Json array result but in this getting mix result with string and equal to and many other things so not able to extract result properly

[dreamer-89]

@dipenpatel235 Can you please share the repro steps along with software sw versions you are using ?

[dipenpatel235]

@dreamer-89

1. I have installed opensearch version 7.10.2.
2. zeek vesion 2.13.0 on client sensor.
3. installed filebeat (7.10.2), and metricbeat(7.10.2) on sensors. and filebeat collect log of zeek and sending to the Elasticsearch
4. created Alert > monitor > write query for generate alert > sending to webhook
5. then received data on webhook that i have show.

in monitor query response get the data in json formate. So, I need a same data as webhook response.

I am using ctx.results to get the data please look at this.

thanks, dipen.

[dblock]

I think this belongs to alerting, moved.

[rishabhmaurya]

@dipenpatel235 can you share your monitor definition?

[dipenpatel235]

@rishabhmaurya I have alredy share all the details of issue. Can you please expain little beat about monitor defination what things you need.

[dipenpatel235]

Hello @dreamer-89 @dblock,

Any update or resolving this issue in OpenSearch version 2.0.1?

[praveensameneni]

Hi @dipenpatel235 , we are looking into the issue, will get back in a couple of days on the research and progress.

[lezzago]

@dipenpatel235, it seems your ask is for ctx.results to be in JSON format. In our documentation, we have listed ctx.results to be an array type. We do not convert it to be a JSON object when its outputted. I recommend submitting a feature request to add support for this.

[qreshi]

Hi @dipenpatel235,

The ctx contents are being stored as a map. So what you're seeing there is the visual representation of a Java map.

I'm not sure why the date formats are messed up (ex. 2022-04-10T08: 31: 49.959Z) since the map is being passed into the arguments for script compilation, it should be using the standard string representation. I don't recall if that's what that format looks like. Could you share the mappings of the non-redacted fields that you've given us examples of? This would help anyone reproduce the data as closely as possible.

I think this is another example of https://github.com/opensearch-project/alerting/issues/59. Unfortunately, Mustache is more of a text templating tool and does not do well for things like JSON templating.

We could look into alternatives in this regard but JSON templating tools may not be as good in the general text templating case. Perhaps having some type of selection option for "this is for text templating/general message being sent" vs. "I need proper JSON here for a webhook payload" could give the best of both worlds in the short term.

[partlov]

@qreshi It seams that space is added after every colon symbol, so I suppose the reason is actually that this is copy paste from some JSON formatter of IDE with formatted JSON. This is also case for IP V6 and MAC addresses.

I agree that we can't do anything special here with if JSON is required from some object in Mustache template. We should probably take some steps to level up scripting and use something which is compatible with Mustache but has some additions for JSON format as this is something required on couple of places so far.

[qreshi]

@partlov Right, I tried including some things I heard of while briefly searching for alternatives here. I'd be interested in seeing comments there on what scripting options people would prefer from those who have some more experience in the different options.

[larsw]

Since there seems to be little progress here; is it possible to have an option to choose between mustache templating for the message or just a raw JSON-serialized version of the context - or to gold-plate it; a JSON-serialized version that can be formatted with JSON Path / jq syntax or something?

Without this, we need to roundtrip back to the API to get the data we're looking for in a structured format - as far as I can tell, it's infeasible to try to parse the current output.

[givilleneuve]

Hello everyone, just checking if we are an idea to when this might be fixed?