Grok - fields with ( ) in them.

[pete-leese]

trafficstars

Hey,

Here is my Grok query but for some reason it cannot find a match when I have the brackets in the Referrer and user agent name.

%{TIMESTAMP_ISO8601:logtime} %{WORD:s-sitename} %{WORD:s-computername} %{IPORHOST:s-ip} %{WORD:cs-method} %{NOTSPACE:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} %{NOTSPACE:cs-version} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Referer)} %{IPORHOST:cs-host} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:c-win32-status} %{NUMBER:sc-bytes} %{NUMBER:cs-bytes} %{NUMBER:time-taken}

Example log item:

2018-02-02 00:01:32 W3SVC1 UKAPPSVR 172.18.131.173 GET /123/I/Home/PLMonstants - 80 Joe+Bloggs 172.18.17.185 HTTP/1.1 Mozilla/5.0+(Windows+NT+6.1;+Trident/7.0;+rv:11.0)+like+Gecko https://blahblah.co.uk/theappname/live/app/thingy localhost 200 0 0 3393 2644 90

was using http://grokconstructor.appspot.com/do/match to validate?

Any ideas what I could be doing wrong or if there is something I can change with the query string to work around the bracket issue.

Unfortunately I cannot change the name of the field as we push into splunk as well.

Thanks.

Pete

[fstab]

In grok_exporter, you need to distinguish two names:

• The names used in the grok pattern (like the name cs_user_agent in %{NOTSPACE:cs_user_agent}) are just used inside the grok_exporter config file, to configure which field ends up in which label. These names are never exposed to Prometheus.
• The label names (like my_own_label in the example below) are presented to Prometheus.

In most cases the same name is used for the grok pattern matches and the Prometheus label names, but it is perfectly ok to use different names like this:

global:
    config_version: 2
input:
    type: file
    path: ./example/test.log
    readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines.
grok:
    patterns_dir: ./logstash-patterns-core/patterns
metrics:
    - type: counter
      name: test_metric_total
      help: Total number of matched lines
      match: '%{TIMESTAMP_ISO8601:logtime} %{WORD:s_sitename} %{WORD:s_computername} %{IPORHOST:s_ip} %{WORD:cs_method} %{NOTSPACE:cs_uri_stem} %{NOTSPACE:cs_uri_query} %{NUMBER:s_port} %{NOTSPACE:cs_username} %{IPORHOST:c_ip} %{NOTSPACE:cs_version} %{NOTSPACE:cs_user_agent} %{NOTSPACE:cs_referer} %{IPORHOST:cs_host} %{NUMBER:sc_status} %{NUMBER:sc_substatus} %{NUMBER:c_win32_status} %{NUMBER:sc_bytes} %{NUMBER:cs_bytes} %{NUMBER:time_taken}'
      labels:
          my_own_label: '{{.cs_user_agent}}'
server:
    host: localhost
    port: 9144

Only the label name my_own_label is presented to Prometheus, the grok pattern name cs_user_agent is only used inside the grok_exporter config file.

There should be no need to use special characters in grok pattern names (like cs_user_agent). If you want to use special characters in the Prometheus label name (like my_own_label): This is unfortunately impossible, because Prometheus label names must match this regular expression ^[a-zA-Z_][a-zA-Z0-9_]*$ (see here). This is a Prometheus restriction and has nothing to do with grok_exporter.

For the label value there are of course no restrictions.

[anirudha86]

I am using Grok exporter to find out error lines. I am able to do that with .log files. However, whenever, I am trying to use the same with XML files, I am not getting desired result.

For Example:- Below is my log file:- 2016/07/30 14:37:03 alice 1.5 30.07.2016 14:37:33 Adam 2.5 30.07.2016 14:43:02 bob 3.5 30.07.2016 14:45:59 Bill 4.5 31.07.2018 14:45:59 Baren 5.5 30.07.2018 14:45:59 Joe 6.5 30.07.2018 14:45:59 Right 7.5 30.07.2018 14:45:59 Jess 8.5 30-07-2018 14:45:59 Nathan 9.5 30.07.2018 15:45:59 Suzi 10.5

Grok Config File:- input: type: file path: ./example/example.log readall: true grok: patterns_dir: ./patterns metrics: - type: counter name: grok_example_lines_total help: Example counter metric with labels. match: '%{DATE} %{TIME} %{USER} %{NUMBER}' server: host: localhost port: 9144

Output:- # TYPE grok_exporter_lines_total counter grok_exporter_lines_total{status="ignored"} 1 grok_exporter_lines_total{status="matched"} 9

Same thing when I am trying to do for xml file, all lines are getting ignored. XML Code:- <Cat>Exception</Cat> <DateTime>2017-02-17T21:12:44.4677331-02:00</DateTime>

Grok Config File:- input: type: file path: ./example/example.xml readall: true grok: patterns_dir: ./patterns metrics: - type: counter name: grok_example_lines_total help: Example counter metric with labels. match: '%{TIMESTAMP_ISO8601}' server: host: localhost port: 9144

In Grok debugger, grok expression is matching with 2017-02-17T21:12:44.4677331-02:00, however, in metrics all lines are getting ignored Output:-

TYPE grok_exporter_lines_total counter

grok_exporter_lines_total{status="ignored"} 5 grok_exporter_lines_total{status="matched"} 0

[fstab]

It seems that the content of your XML example got lost in the comment. Could you comment it again?

The best way to comment an XML snippet on Github is to use three backticks ``` at the begin and the end, like this:

```
your xml example here
```

[anirudha86]

Will Try Again :)

I am using Grok exporter to find out error lines. I am able to do that with .log files. However, whenever, I am trying to use the same with XML files, I am not getting desired result.

For Example:- Below is my log file:-

  2016/07/30 14:37:03 alice 1.5
    30.07.2016 14:37:33 Adam 2.5
    30.07.2016 14:43:02 bob 3.5
    30.07.2016 14:45:59 Bill 4.5
    31.07.2018 14:45:59 Baren 5.5
    30.07.2018 14:45:59 Joe 6.5
    30.07.2018 14:45:59 Right 7.5
    30.07.2018 14:45:59 Jess 8.5
    30-07-2018 14:45:59 Nathan 9.5
    30.07.2018 15:45:59 Suzi 10.5

Grok Config File:-

    input:
        type: file
        path: ./example/example.log
        readall: true
    grok:
        patterns_dir: ./patterns
    metrics:
        - type: counter
          name: grok_example_lines_total
          help: Example counter metric with labels.
          match: '%{DATE} %{TIME} %{USER} %{NUMBER}'
    server:
        host: localhost
        port: 9144

Output:-

    # TYPE grok_exporter_lines_total counter
    grok_exporter_lines_total{status="ignored"} 1
    grok_exporter_lines_total{status="matched"} 9

Same thing when I am trying to do for xml file, all lines are getting ignored. XML Code:-

   <Cat>Exception</Cat>
          <DateTime>2017-02-17T21:12:44.4677331-02:00</DateTime>

Grok Config File:- input:

    type: file
    path: ./example/example.xml
    readall: true
grok:
    patterns_dir: ./patterns
metrics:
    - type: counter
      name: grok_example_lines_total
      help: Example counter metric with labels.
      match: '%{TIMESTAMP_ISO8601}'
server:
    host: localhost
    port: 9144

In Grok debugger, grok expression is matching with 2017-02-17T21:12:44.4677331-02:00, however, in metrics all lines are getting ignored Output:-

# TYPE grok_exporter_lines_total counter
grok_exporter_lines_total{status="ignored"} 5
grok_exporter_lines_total{status="matched"} 0

[fstab]

This is because with the current version v0.2.5 the match must match the entire line, not just part of the line. If you change it to '.*%{TIMESTAMP_ISO8601}.*' it should work.

The current version is unexpected behaviour. I changed it in the master branch some time ago. With the next release, the match counts if any part of the line matches, so you will not need the .* with the next release. I realized it has been a while since the last release, so I will release v0.2.6 soon. With v0.2.6 your example should work as is.