Need fluentd parsing for NGINX NAXSI Logs

[aarvee11]

Hi All,

We need to parse NAXSI_FMT and NAXSI_EXLOG logs to be parsed by fluentd which shall be further shipped into elasticsearch. We do not want to build logstash for just parsing the logs using GORK patterns instead use fluentd's excellent capability in performing the same.

A sample log is given below:

2019/01/16 07:34:06 [error] 46#0: *17 NAXSI_FMT: ip=172.17.0.1&server=api.secops.in&uri=/v1/collect/pay_BkdDYXLVTVzuG10436/callback/3cbca6c56156bcad278025c450e7551b1e467bd9/secops_live_ILgsfZjhaskjfhZoFIKMb&vers=0.56&total_processed=3&total_blocked=3&config=learning&cscore0=$SQL&score0=492&cscore1=$XSS&score1=472&zone0=HEADERS&id0=1001&var_name0=cookie&zone1=HEADERS&id1=1010&var_name1=cookie&zone2=HEADERS&id2=1015&var_name2=cookie&zone3=HEADERS&id3=1310&var_name3=cookie&zone4=HEADERS&id4=1311&var_name4=cookie&zone5=HEADERS&id5=1315&var_name5=cookie&zone6=BODY&id6=1002&var_name6=pares&zone7=BODY&id7=1009&var_name7=pares, client: 172.17.0.1, server: , request: "POST /v1/collect/pay_BkdDYXLVTVzuG10436/callback/3cbca6c56156bcad278025c450e7551b1e467bd9/rzp_live_ILgsfZCZoFIKMb HTTP/1.1", host: "api.secops.in", referrer: "https://xxx.yyyyyyyy.com/xxx-web-v2?ISSUER_ID=8117"

For further reading please refer to NAXSI_Logs_Documentation

[cosmo0920]

@okkez Could you take a look? I guess that we can handle this log with fluent-plugin-grok-parser .

[okkez]

@aarvee11 You can use grok patterns with fluent-plugin-grok-parser if you want to parse your logs using grok patterns. But built-in grok patterns cannot parse NAXSI_FMT and NAXSI_EXLOG logs. You can add custom grok patterns to parse your logs, see fluent-plugin-grok-parser's document for more details.

Or you can use built-in parser_regexp plugin.