Help with gork_exporter

[sheevaeva]

trafficstars

Hello

I'm using grok exporter and here is what I want to achieve: I have a Java application whose log entry is in below format:

{"@version":1,"source_host":"fstest-stage-bm-62","message":"Known host file not configured, using user known host file: /home/.ssh/known_hosts","thread_name":"Camel (camel-1) thread #4 - aws-s3://fstest-stage-bm-62","@timestamp":"2019-08-28T07:52:12.526+00:00","level":"INFO","logger_name":"org.apache.cam.file.remote.oerations"}

I want to configure Prometheus alert for any 'ERROR' entry in the log level. Here is how the grok_exporter config.yml file look like:

global: config_version: 2 input: type: file path: ./example/test.log readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines. grok: patterns_dir: ./patterns

metrics: - type: counter name: error_test help: Counter metric example match: '%{NUMBER} %{JAVACLASS} %{JAVALOGMESSAGE} %{JAVATHREAD} %{TOMCAT_DATESTAMP} %{LOGLEVEL:severity} %{JAVAMETHOD}' labels: grok_field_name: severity prometheus_label: severity server: host: 0.0.0.0 port: 9144

============================ The test log file has 4 log lines with one log line having log level as ERROR. I did try accessing http://IP:9144/metrics and I see the below but there is no metric created on Prometheus(grok_exporter is installed on Prometheus itself).

grok_exporter_line_processing_errors_total{metric="error_test"} 0

HELP grok_exporter_lines_matching_total Number of lines matched for each metric. Note that one line can be matched by multiple metrics.

TYPE grok_exporter_lines_matching_total counter

grok_exporter_lines_matching_total{metric="error_test"} 0

HELP grok_exporter_lines_processing_time_microseconds_total Processing time in microseconds for each metric. Divide by grok_exporter_lines_matching_total to get the average processing time for one log line.

TYPE grok_exporter_lines_processing_time_microseconds_total counter

grok_exporter_lines_processing_time_microseconds_total{metric="error_test"} 0

HELP grok_exporter_lines_total Total number of log lines processed by grok_exporter.

TYPE grok_exporter_lines_total counter

grok_exporter_lines_total{status="ignored"} 4 grok_exporter_lines_total{status="matched"} 0

I do see the metric on prometheus but ti doesn't yield any value. Can someone please help me with regex expression for my json log format as I couldn't get the correct matching format.

Thanks

[fstab]

There's something wrong with your match configuration, because it does not fit together with the format of your log line. If you just want to count the number of ERROR messages, the simplest thing you can do is as follows:

global:
  config_version: 2
input:
  type: file
  path: ./example/test.log
  readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines.
grok:
  patterns_dir: ./patterns
metrics:
- type: counter
  name: error_test
  help: Counter metric example
  match: '"level":"ERROR"'
server:
  host: 0.0.0.0
  port: 9144

[sheevaeva]

Thank you Fabian for the response. The metric does match with the data I have added in the log file but i do not see the metric on Prometheus.

grok_exporter_lines_total{status="ignored"} 1 grok_exporter_lines_total{status="matched"} 1

I also tried assigning a label "severity" to the match string but that does not work either. Can you please take a look.

global: config_version: 2 input: type: file path: ./example/test.log readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines. grok: patterns_dir: ./patterns metrics: - type: counter name: total_error help: Counter metric example match: '"level":"ERROR"':severity labels: grok_field_name: severity prometheus_label: severity server: host: 0.0.0.0 port: 9144

Thanks

On Fri, Sep 20, 2019 at 3:07 AM Fabian Stäber [email protected] wrote:

There's something wrong with your match configuration, because it does not fit together with the format of your log line. If you just want to count the number of ERROR messages, the simplest thing you can do is as follows:

global: config_version: 2input: type: file path: ./example/test.log readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines.grok: patterns_dir: ./patternsmetrics:

• type: counter name: error_test help: Counter metric example match: '"level":"ERROR"'server: host: 0.0.0.0 port: 9144

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/fstab/grok_exporter/issues/70?email_source=notifications&email_token=ANHIUPCXENY53CNWSDU2CPLQKPWINA5CNFSM4IX2CK3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7E4XTI#issuecomment-533318605, or mute the thread https://github.com/notifications/unsubscribe-auth/ANHIUPEYZMMNR2UR44NII73QKPWINANCNFSM4IX2CK3A .

[fstab]

i do not see the metric on Prometheus

If grok_exporterexposes the metric but you don't see it in Prometheus, then something with your Prometheus config is wrong. Could you post your Prometheus config?

And please, for better formatting in GitHub put a line with three ` characters before your yaml and a line with three ` characters after your yaml like this:

```
yaml here
```

[sheevaeva]

Here is how my preometheus config looks like :

 - job_name: 'grok'
    metrics_path: /grok
    static_configs:
      - targets:
          - IP:9144
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9144 # This exporter's real hostnane

[fstab]

Try removing metrics_path: /grok.

[sheevaeva]

Thanks Fabian, i will try this our shortly! I also need the "message" to be printed in case of "ERROR". How do i get this done in the grok config?

[fstab]

I think it's best to approach this step by step. Let's first try to see the metric in Prometheus without any labels, and then see where we can go from there. There are a few things to consider when using error messages as label values, such as cardinality explosion. You'll need to be careful with that.

[sheevaeva]

Thanks Fabian. I tried what you suggested and I do see the metric on Prometheus now. We now need to work on sending 'message' in the alert that is received. Appreciate your help.

Regards!

On Tue, Sep 24, 2019 at 2:43 AM Fabian Stäber [email protected] wrote:

I think it's best to approach this step by step. Let's first try to see the metric in Prometheus without any labels, and then see where we can go from there. There are a few things to consider when using error messages as label values, such as cardinality explosion. You'll need to be careful with that.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/fstab/grok_exporter/issues/70?email_source=notifications&email_token=ANHIUPH3KTSF5V25QDNIRMDQLEWOPA5CNFSM4IX2CK3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7MJGCQ#issuecomment-534287114, or mute the thread https://github.com/notifications/unsubscribe-auth/ANHIUPGBHPLRMBXKATW6UPDQLEWOPANCNFSM4IX2CK3A .

[sheevaeva]

Hi Fabian

Any update here please?

Thanks!

[fstab]

If you really want to use the error message as a label value you can do it like this:

global:
  config_version: 2
input:
  type: file
  path: ./test.log
  readall: true
grok:
  patterns_dir: ./patterns
  additional_patterns:
  - 'MESSAGE [^"]*'
metrics:
- type: counter
  name: error_test
  help: Counter metric example
  match: '(?=.*"level":"ERROR").*"message":"%{MESSAGE:message}"'
  labels:
    message: '{{.message}}'
server:
  host: 0.0.0.0
  port: 9144

However, you will get performance issues if you have many different log messages, because internally the Prometheus server will create a new time series for each message. Google "cardinality explosion" to learn more about this issue.

[sheevaeva]

Understood, Thanks Fabian.

[sheevaeva]

Hi Fabian, a quick question. Can you let me know if grok exporter is compatible with Prometheus version 2.3.2? It doesn't seem to working for me while it is fine with Prometheus version 2.9.1

[sheevaeva]

Hi Fabian Can you please let me knowif grok exporter is compatible with Prometheus version 2.3.2? It doesn't seem to be working fine for me , specially with alerting rules while it is fine with Prometheus version 2.9.1

[fstab]

Yes, it's compatible. There are no breaking changes between these Prometheus versions.

Alerting rules have nothing to do with grok_exporter anyway, grok_exporter is only focused on providing metrics, but not on alerting.

[xuanyuanaosheng]

@fstab Could you please add a grafana template? So we can use it, you can collect some basic log metrics like system log.