After upgrading to V1.0.1 receiving client timeout error

[siddharthajuprod07]

Hi, We have seen an issue where the older version of prometheus app is working fine but the CPU usage was very high (80%). So we upgarded the app to recent version but after the metrices stopped flowing to splunk and we are receiving client timeout error. Can someone please throw some light on this.

[lukemonahantnt]

I don't believe any changes in this time period should be increasing or decreasing CPU usage in any manner.

For the rest of it, I'm not sure if I can help without more logs or configuration details. Is there anything you can provide here?

[siddharthajuprod07]

@lukemonahantnt , Thank you for your reply, please find the log errors which we are seeing, 8/22/24 3:33:30.562 PM 08-22-2024 15:33:30.562 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:33:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:32:30.601 PM 08-22-2024 15:32:30.601 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:32:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:31:30.486 PM 08-22-2024 15:31:30.486 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:31:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:30:30.503 PM 08-22-2024 15:30:30.503 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:30:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:29:30.501 PM 08-22-2024 15:29:30.501 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:29:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:28:30.455 PM 08-22-2024 15:28:30.455 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:28:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:27:34.982 PM 08-22-2024 15:27:34.982 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheusrw" 2024/08/22 14:27:34 Prometheus input globally disabled host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd 8/22/24 3:27:31.005 PM 08-22-2024 15:27:31.005 +0100 ERROR ExecProcessor [3415907 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus" 2024/08/22 14:27:30 net/http: request canceled (Client.Timeout exceeded while reading body) host = source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

again many thanks for your time.

[siddharthajuprod07]

This is the current config we have

[prometheus://test] URI = http://host:port/federate match = {name=~"..*"} index = <index_name> sourcetype = <sourcetype_name> interval = 60 disabled = 0 #end of change