matcher: support metadata grok semantic/identifier

[dchsueh]

logstash supports using a metadata semantic/identifier in grok (see https://www.elastic.co/blog/logstash-metadata "Date filter")

e.g. pattern %{TIMESTAMP_ISO8601:[@metadata][ts]} should match input 2021-03-08 23:20:36.952

but instead reports no match

Thank you.

[stoerr]

Right, sorry - that's a newer feature of grok that isn't implemented here so far. I don't quite understand that - would be the right thing there to just ignore the :... part and treat that as %{TIMESTAMP_ISO8601}?

[christianhaessler]

In my opinion the part after the ":" ist the field name. In this case logstash use this syntax to create a nested field { "@metadata" { "ts": 2021-03-08 23:20:36.952 }

The easiest way to support this in your application would be to create a filed separated with "." like @metadata.ts

I've the same issue with this logline: <166>Feb 25 12:52:25 libfw[3798]: [NOTIFICATION_TEST]Sending a Test Notification.

In Logstash i use this grokfilter: ^<%{POSINT:[log][syslog][priority]:int}>%{SPACE}%{SYSLOGTIMESTAMP:syslog_timestamp}%{SPACE}%{GREEDYDATA:message}

I can not thes this in your matcher. Therefore i replace the nested field [log][syslog][priority] whith a placeholder for testing ^<%{POSINT:placeholder:int}>%{SPACE}%{SYSLOGTIMESTAMP:syslog_timestamp}%{SPACE}%{GREEDYDATA:message}

Would be really nice if it would work out of the box 👍

[stoerr]

Ah, OK, I have overlooked something in https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_grok_basics :

The syntax for a grok pattern is %{SYNTAX:SEMANTIC}

The SYNTAX is the name of the pattern that will match your text. For example, 3.44 will be matched by the NUMBER pattern and 55.3.244.1 will be matched by the IP pattern. The syntax is how you match.

The SEMANTIC is the identifier you give to the piece of text being matched. For example, 3.44 could be the duration of an event, so you could call it simply duration. Further, a string 55.3.244.1 might identify the client making a request.

Optionally you can add a data type conversion to your grok pattern. By default all semantics are saved as strings. If you wish to convert a semantic’s data type, for example change a string to an integer then suffix it with the target data type. For example %{NUMBER:num:int} which converts the num semantic from a string to an integer. Currently the only supported conversions are int and float.

So, it seems that I indeed should ignore everything after the : until the next } since neither that that identifier matters when I check for a match, nor does that data type conversion.

[stoerr]

OK, I changed it. Unfortunately I had to make a little hack, since my joni regex version doesn't support group names starting with [ , but does support group names like [log][syslog][priority] if I prefix it with a _ and show it as _[log][syslog][priority] in the matcher. Hopefully people will know what I mean. :-) That's already live now.

[christianhaessler]

Great. Worked like a charm

Thanks a lot.