splunk_modinput_prometheus icon indicating copy to clipboard operation
splunk_modinput_prometheus copied to clipboard

Splunk Base App

Open rafeeqsid opened this issue 2 years ago • 3 comments

Working on to integrate metric forwarding via Prometheus remote-write to Splunk HF.

Current Setup:- Splunk HF is hosted on HTTPS. As below document suggest for HTTP, Than how to use HTTPS for remote-write?

Using Below configuration in Splunk-HF as reference to https://splunkbase.splunk.com/app/4077

[prometheusrw] port = 8098 maxClients = 10 disabled = 0

[prometheusrw://testing] bearerToken = ABC123 index = prometheus whitelist = * sourcetype = prometheus:metric disabled = 0

And not able to justify the team that how to configure bearer-token in prometheus, As for this no document is available. And also how to use HTTPS in url as splunk hosted over HTTPS

remote_write: - url: "http://myhost:8098" bearer_token: "ABC123" write_relabel_configs: - source_labels: [name] regex: expensive.* action: drop

rafeeqsid avatar Apr 16 '23 23:04 rafeeqsid

To enable HTTPs on the Splunk side, please use the "enableTLS" and other related options in inputs.conf.

https://github.com/lukemonahan/splunk_modinput_prometheus/blob/master/modinput_prometheus/README/inputs.conf.spec#L11-L18

On the Prometheus side it should be just using a "https" URL, and ensuring the tls_config is set to trust the cert you have provided within Splunk.

It appears like Prometheus has changed config for bearer tokens in more recent versions. From reading docs, it seems this might be correct, but I haven't tested it. Will get onto this and update the Splunk add-on docs when I can.

authorization:
    type: "Bearer"
    credentials: "ABC123"

tls_config and new Bearer token options described here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

lukemonahantnt avatar Apr 16 '23 23:04 lukemonahantnt

Yes above suggestion was very helpful.

One more query for below inputs.conf for remote-write integration the index(prometheus) will be metric type or Event type?

[prometheusrw://testing] bearerToken = ABC123 index = prometheus whitelist = * sourcetype = prometheus:metric disabled = 0

rafeeqsid avatar Apr 19 '23 11:04 rafeeqsid

Use a metrics type index in all cases.

lukemonahantnt avatar Apr 19 '23 23:04 lukemonahantnt