fluent-plugin-grok-parser icon indicating copy to clipboard operation
fluent-plugin-grok-parser copied to clipboard

nginx logs are not parsed using grok pattern

Open jatinder10884 opened this issue 5 years ago • 2 comments

I am using below grok_pattern for parsing Nginx logs

<source>

  @type tail
  path /data/nginxlogs/access.log
  pos_file /var/log/td-agent/nginx-access.log.pos
  tag es.nginx.acces
  <parse>
   @type multiline_grok
   <grok>
    pattern Started %{IPORHOST:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:server_response} %{NUMBER:bytes} \"%{URI:referrer}\" %{QS:user_agent}
#   grok_pattern %{MY_SUPER_PATTERN}
#   custom_pattern_path /etc/td-agent/MY_SUPER_PATTERN
#   multiline_start_regexp /^[^\s]/
  </grok>
  keep_time_key true
  </parse>
</source>
<filter es.nginx.acces>
  @type record_transformer
  <record>
  hostname ${hostname}
  </record>
</filter>
<match es.**>
  type "aws-elasticsearch-service"
  type_name "access_log"
  logstash_format true
 logstash_prefix "ngrokcontainer"
include_tag_key true
  tag_key "@log_name"
  flush_interval 1s
<endpoint>
    url http://testing-logs-ab-east-1.es.amazonaws.com
    region us-east-1
  </endpoint>
</match>

Kibana :

message 3.231.213.28 - - [18/Apr/2020:22:20:16 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
Its not parsing properly, but when I check this grok debugger its working fine.
Please let me know how I can fix this issue

jatinder10884 avatar Apr 18 '20 22:04 jatinder10884

Could you show me your nginx.conf around log formatting?

And consider using https://docs.fluentd.org/parser/nginx, please.

okkez avatar May 30 '20 02:05 okkez

We recommend to use built-in parser as much as possible.

<source>
  @type tail
  path /data/nginxlogs/access.log
  pos_file /var/log/td-agent/nginx-access.log.pos
  tag es.nginx.acces
  <parse>
    @type nginx
    keep_time_key true
  </parse>
</source>

is better.

cosmo0920 avatar Sep 17 '20 07:09 cosmo0920