Allow for multiple inputs defined in a single grok_exporter process

[hartfordfive]

trafficstars

Currently, it appears it's only possible to have a single input source, which can be one of webhook, file or stdin. Unfortunately, this is problematic in cases where multi-tenancy is a factor. For example, say you have a large VM or bare metal host which hosts 10 or more different applications. If the owners of the individual applications each want to monitor for specific strings in their logs, we would need to spin up 10 different grok exporters on this host to monitor each of the 10 log files. Although this is more reasonable when it comes to Kubernetes, as this could simply run as a sidecar to the pod which is generating the logs, this doesn't work out so well in a non-containerized scenario.

It would be great if multiple inputs could be specified, allowing each input to specify the log file path, grok patterns, and resulting metrics. To give a better idea of what I envision this could resemble, here's an example updated config

global:
  config_version: 4
  retention_check_interval: 53s
inputs:
  - 
    type: file
    paths: 
      - /apps/logs/app1/output.log
    readall: false 
    fail_on_missing_logfile: false 
    poll_interval: 10
    imports:
    - 
      type: grok_patterns
      dir: /etc/grok_exporter/patterns
    metrics:
    - 
      type: counter
      name: oom_errors
      help: The number of message counts contain the specific strings.
      match: 'out of memory'
      labels:
        logfile: '{{.logfile}}'
  - 
    type: file
    paths: 
      - /apps/logs/app2/output.log
    readall: false 
    fail_on_missing_logfile: false 
    poll_interval: 10
    imports:
    - 
      type: grok_patterns
      dir: /etc/grok_exporter/patterns
    metrics:
    - 
      type: counter
      name: app_errors
      help: The number of message counts contain the specific strings.
      match: 'foobar error'
      labels:
        logfile: '{{.logfile}}'

server:
  host: 0.0.0.0
  port: 9144

I haven't had the chance to examine the code in depth so I admit I'm unsure on the degree of complexity of these changes. It may require a fair amount of modifications to the existing code base, although I think other users may also appreciate this added feature.

[fstab]

I'm not sure about this. Monitoring multiple log files is already possible. The only thing that doesn't work right now is mixing logfiles with webhooks. So if you really have both, the worst case would be that you start two grok_exporter instances, one for logfiles and one for webhooks.

[yurenchen000]

@fstab maybe @hartfordfive 's meaning, declare metrics under file scope. rather than declare file limit under each metrics scope repeatedly.

[yurenchen000]

first i want to write this, variables

config.yml

metrics:
- type: gauge
  path: ${cloud_log}
  name: call_send_total
  help: Total number of call send.
  match: '%{CL_TIME} CALL_SEND=%{INT:call_send}'
  value: '{{.call_send}}'

- type: gauge
  path: ${cloud_log}
  name: call_recv_total
  help: Total number of call recv.
  match: '%{CL_TIME} CALL_RECV=%{INT:call_recv}'
  value: '{{.call_recv}}'

Unfortunately, variable ${xxx} or {{xxx}} not support.

then, I use this workaround, imports

config.yml

input:
  type: file
  path: /home/log/cloud*.out
  readall: false
imports:
- type: grok_patterns
  dir: ./patterns
- type: metrics
  file: ./cloud_metrics.yml
  defaults:
    path: /home/log/cloud.out  # it should contain in gloabl.path, otherwise seems not work

./cloud_metrics.yml

- type: gauge
  name: call_send_total
  help: Total number of call send.
  match: '%{CL_TIME} CALL_SEND=%{INT:call_send}'
  value: '{{.call_send}}'

- type: gauge
  name: call_recv_total
  help: Total number of call recv.
  match: '%{CL_TIME} CALL_RECV=%{INT:call_recv}'
  value: '{{.call_recv}}'

this seems worked: limit metrics under file scope, calmly, no repeatedly

still hope the file scope metrics, or

• variables support rather than repeatedly file path in each metrics, or
• support multi top-level config.yml (each has independent input.path scope) in --config args