Possible incompatibility with grok parser's grok XML elements

[r4j4h]

Hi,

Thanks for the great plugin!

We may be attempting to configure it improperly esp. around the v0.14 changes, but are at a loss as to how to use <grok> elements from the grok plugin with this plugin?

For reference, the basic grok_pattern works well with separate individual <pattern>format grok</pattern>s.

Fluent Version: 0.14.14

Given the following config block:

<filter **>
  @type parser
  key_name log
  reserve_data false
  <parse>
    @type multi_format
    <pattern>
        format grok
        <grok>
          # e.g. 2017-04-10 08:07:49.062 DEBUG [doot-boop,d0ea3905d59025c2,d0ea3905d59025c2,false] 17206 --- [  XNIO-2 task-5] io.undertow.session                      : Created session ... snip ...
          pattern %{TIMESTAMP_ISO8601:time}%{SPACE}%{LOGLEVEL:level}%{SPACE}\[%{DATA:app_name},%{DATA:trace_id},%{DATA:span_id},%{DATA:export_to_zipkin}\]%{SPACE}%{NUMBER:pid}%{SPACE}---%{SPACE}\[%{SPACE}%{DATA:threadname}\]%{SPACE}%{JAVACLASS:classname}%{SPACE}:%{SPACE}%{GREEDYDATA:logmessage}
        </grok>
        <grok>
          # e.g. 2017-03-27 22:15:59.861 DEBUG 5 --- [t-dispatcher-37] com.acme.aspects.LogAspect              : snip\n
          pattern %{GREEDYDATA:logmessage}
        </grok>
    </pattern>
    <pattern>
        format grok
    </pattern>
  </parse>
</filter>

When starting fluent we get:

2017-04-11 22:22:13 +0000 [info]: adding filter pattern="**" type="parser"
2017-04-11 22:22:13 +0000 [info]: #0 Expanded the pattern  into (?<time>(?>\d\d){1,2}-(?:0?[1-9]|1[0-2])-(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])[T ](?:2[0123]|[01]?[0-9]):?(?:[0-5][0-9])(?::?(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))?(?:Z|[+-](?:2[0123]|[01]?[0-9])(?::?(?:[0-5][0-9])))?)\s*(?<level>([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))\s*\[(?<app_name>.*?),(?<trace_id>.*?),(?<span_id>.*?),(?<export_to_zipkin>.*?)\]\s*(?<pid>(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))))\s*---\s*\[\s*(?<threadname>.*?)\]\s*(?<classname>(?:[a-zA-Z0-9-]+\.)+[A-Za-z0-9$]+)\s*:\s*(?<logmessage>.*)
2017-04-11 22:22:13 +0000 [info]: #0 Expanded the pattern  into (?<logmessage>.*)
2017-04-11 22:22:13 +0000 [error]: #0 config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="no grok patterns. Check configuration, e.g. typo, configuration syntax, etc"

[ramya397]

hi r4j4h

i am facing the same issue from 3 days if you get solution for this kindly help me .

Ramya

[unquietwiki]

I was banging my head on this all evening. Turns out when you install the parser plugin, it defaults to newer version & loads td-agent 0.14.20. So if you want a working version, you have to stay on 0.12.x and force load an older version of the plugin.

td-agent-gem uninstall fluent-plugin-grok-parser --version=2.1.4 td-agent-gem install fluent-plugin-grok-parser --version=1.0.0 td-agent-gem install fluentd --version=0.12.39

[repeatedly]

Sorry for too late response. First case, the problem is second <pattern>. This doesn't have <grok> element and it causes "no grok patterns". Adding <grok> to second <pattern> avoids the problem.