%cpu starts at 0.3% and gradually builds up, reaching 100% in few days

[pon-raj]

trafficstars

Hi,

I am using “grok exporter” to parse apache web server access logs. It is running on RHEL 7. The cpu of the process starts from 0.3% and gradually increases over the days. In few days, it hits 100% and I have to kill it.

Please help to figure out the reason for this gradual cpu build up.

config.yml: global: config_version: 3 input: type: file paths:

• /logs/access_log.* readall: false fail_on_missing_logfile: false imports:
• type: grok_patterns dir: /grok_exporter/patterns grok_patterns:
• 'BOOLEAN True|False' metrics:
• type: gauge name: apache_http_request help: HTTP requests to Apache match: '%{APACHE}' labels: namespace: 'apache' client_ip: '{{.client_ip}}' forwarded_ip: '{{.forwarded_ip}}' timestamp: '{{.timestamp}}' method: '{{.method}}' uri: '{{.resource}}' code: '{{.response}}' time: '{{.responsetime}}' size: '{{.bytes}}' vhost: '{{.vhost}}' vip: '{{.vip}}' backend: '{{.backend}}' pipeline: '{{.pipeline}}' value: '{{timestamp "02/Jan/2006:15:04:05 -0700" .timestamp}}'

server: protocol: http port: 9144

grok_exporter_build_info{branch="master",builddate="2020-07-07",goversion="go1.14.4",instance="wlpp1",job="apache",platform="linux-amd64",revision="02cb3df",version="1.0.0.RC4"} grok_exporter_lines_total{instance="wlpp1",job="apache",status="ignored"} 0 grok_exporter_lines_total{instance="wlpp1",job="apache",status="matched"} 18409 grok_exporter_lines_matching_total{instance="wlpp1",job="apache",metric="apache_http_request"} 18409 grok_exporter_line_buffer_load{instance="wlpp1",interval="1m",job="apache",value="max"} 1 grok_exporter_line_buffer_load{instance="wlpp1",interval="1m",job="apache",value="min"} 0 grok_exporter_line_processing_errors_total{instance="wlpp1",job="apache",metric="apache_http_request"} 0 grok_exporter_lines_processing_time_microseconds_total{instance="wlpp1",job="apache",metric="apache_http_request"} 19636139

Pattern to match: (?:%{IP:client_ip}|-)? (?:%{IP:forwarded_ip}|-)? (?:%{NOTSPACE:remote_login}|-)? (?:%{NOTSPACE:remote_user}|-)? [%{HTTPDATE:timestamp}] "%{WORD:method} (?:%{URIPATH:resource})?([file://%3f(%3f:%25%7bNOTSPACE:parm%7d)]\?(?:%{NOTSPACE:parm}))? (HTTP/%{NUMBER:version}")? (%{NUMBER:response})? (?:%{NUMBER:bytes}|-) (?:%{NUMBER:responsetime}|-)? (?:%{NOTSPACE:global_transactiond_id}|-)? (?:%{NOTSPACE:unique_id}|-)? (?:%{WORD:vhost}|-)?-(?:%{WORD:pipeline}|-)? (?:"%{NOTSPACE:backend}"|-)? (?:"%{NOTSPACE:vip}"|-)? ("%{DATA:http_request_referrer}"|-)? ("%{DATA:user_agent_original}"|-)? "(%{DATA:cookie}"|-)?

Log data: 10.9.162.9 10.8.168.254 - - [17/Jul/2020:19:59:59 -0400] "GET /api/v1/vehicles/12500099529/synch HTTP/1.1" 200 112 12231 cf80eed0eed75f123b7f0821f2af XxI7fwoIWo2khXmIAABHWo2kAAAOW vehicles_restsvc-test "hots1:9096" "veh.org.net" "-" "axios/0.19.2" "-\ 10.9.162.9 10.8.168.254 - - [17/Jul/2020:19:59:59 -0400] "GET /api/v1/vehicles/12500099529/synch HTTP/1.1" 200 112 12231 cf80eed0eed75f123b7f0821f2af XxI7fwoIWo2khXmIAABHWo2kAAAOW vehicles_restsvc-test "hots1:9096" "veh.org.net" "-" "axios/0.19.2" "-"

Thanks.

[fstab]

This is weird, CPU usage should be constant. Did you see any increase in memory usage?

[fstab]

I tried to reproduce the behavior with your data, but something's wrong with the pattern you posted above:

[%{HTTPDATE:timestamp}]

Square brackets are special characters in regular expressions. If your timestamp is in literal square brackets, you need to escape the bracket like this:

\[%{HTTPDATE:timestamp}]

I guess there was a copy-and-paste error when you opened this issue. The pattern above results in an error on grok_exporter startup.

Could you please double check and post your actual configuration? I'd like to reproduce this issue, and it would be helpful to have exactly the same pattern you are using.

[fstab]

Just as a side note: I'm working on improving the built-in metrics to make it easier to see where grok_exporter spends processing time. This will be part of the next release and hopefully make it more obvious where CPU usage comes from.