[prometheus-kube-stack] Https not working under AddtitionalScrapeConfigs in prometheusSpec

[VipulZopSmart]

Describe the bug a clear and concise description of what the bug is.

Tried to use promtheus.prometheusSpec.AdditionalScapeConfigs from kube-prometheus-stack helm chart to scrape confluent cluster metrics to local prometheus. Not able to scrape, getting some certificate error. Also, tried to exec into that pod and tried to wget to the rest endpoint, not able to reach out to any endpoint starting with https.

What's your helm version?

x509, certificate signed by unknown authority

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:33:37Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"darwin/arm64"}

Which chart?

kube-promtheus-stack

What's the chart version?

39.11.0

What happened?

Tried to use promtheus.prometheusSpec.AdditionalScapeConfigs from kube-prometheus-stack helm chart to scrape confluent metrics to local prometheus.

but it's giving error

What you expected to happen?

How to reproduce it?

No response

Enter the changed values of values.yaml?

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: https://xyzapi/v1/push
        headers:
          X-Scope-OrgID: dev
    additionalScrapeConfigs:
      - job_name: Confluent Cloud
        static_configs:
          - targets:
            - api.telemetry.confluent.cloud
        scheme: https
        basic_auth:
          username: abc
          password: def
        metrics_path: /v2/metrics/cloud/export
        params:
          "resource.kafka.id":
            - lkc-abc

Enter the command that you execute and failing/misfunctioning.

minikube start
helm install prometheus prometheus-community/kube-prometheus-stack
helm upgrade -f val.yaml prometheus prometheus-community/kube-prometheus-stack
kubectl --namespace default port-forward svc/prometheus-kube-prometheus-prometheus 9090

Anything else we need to know?

No response

[AndrStnz]

In your scrape job you need to add

job_name: <....>
scheme: https
tls_config:
  ca_file: <path_to_ca_file>

If configured with https prometheus expects a trusted certificate so you have to provide a valid CA prometheus can check against.

[VipulZopSmart]

@AndrewwHummer thanks for the suggestion. Will try this out.

[VipulZopSmart]

Tried to give a certificate, it's not even showing the added job now.

[AndrStnz]

If you job is disappering this usually means the configuration is invalid. Look into the logs of the prometheus pod, or the operator pod. Check if your ca path is correct, check for obvious indent errors.

[VipulZopSmart]

As you can see above, i've used the Confluent Cloud job as they have given in the confluent documentation. Also, the ca path is correct that i've given and no indentation error.

[VipulZopSmart]

I've exec into that pod and provided that certificate, then it worked. But while i would be using with my env clusters there i dont have access to those pods, and i can't add this certificate manually there. Can someone help how to add local certificate file in tls_config for this https job.

[zeritti]

A quick, but not the only, way would be using values' field prometheusSpec.secrets [0] to obtain a mount of the secret's key's value at /etc/prometheus/secrets/SECRET_NAME/KEY in the Prometheus container. You can then set this file path in the Prometheus' job configuration:

1. Create a secret in the chart's namespace. E.g. secret ca-certificates with a CA certificate chain at key ca.pem.
2. Add this secret's name in your chart's values:

prometheus:
  prometheusSpec:
    secrets:
      - ca-certificates

The secret must exist before the chart is rolling out. The key's value will eventually get mounted at /etc/prometheus/secrets/ca-certificates/ca.pem. 3. Set this path in the job confguration through tls_config.ca_file

[0] prometheusSpec.secrets in values

[AlexDaichendt]

Hey!

I am facing a (similar) issue. It appears like tls_config.insecure_skip_verify and/or the ca file are ignored. I keep getting x509: certificate signed by unknown authority even though I provide the correct ca file, which I can verify by using curl locally to my application with the ca.pem. I followed the suggestion by @zeritti from the previous post:

- job_name: netbox_node_exporter
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  authorization:
    type: Token
    credentials: <secret>
  tls_config:
    ca_file: /etc/prometheus/secrets/ca-certificates/ca.pem
    insecure_skip_verify: true
  follow_redirects: true
  enable_http2: true
  relabel_configs: [.....]
  http_sd_configs:
  - follow_redirects: true
    enable_http2: true
    refresh_interval: 1m
    url: https://netbox.cpmpany.loc/api/plugins/prometheus-sd/virtual-machines?status=active

If I look at the certificate, that is mounted inside the container, I can also confirm that it is the correct one. It does not matter, if I set insecure_skip_verify or not; also the error does not change regardless of what I place in ca_file. I am certain, that the config is applied since I copied the previous snippet from the prometheus web ui configuration page.

On the mailing list I read something about Go ignoring some parts of the ca certificate. In case it's relevant, this is the output of openssl x509 -in ca.pem --text :

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = DE, ST = Bavaria, L = location, O = companyname, emailAddress = [email protected], CN = COMPANY HA-002 Beta1 CA, OU = Operating
        Validity
            Not Before: Mar 27 13:05:53 2017 GMT
            Not After : Mar 25 13:05:53 2027 GMT
        Subject: C = DE, ST = Bavaria, L = location, O = companyname, emailAddress = [email protected], CN = COMPANY HA-002 Beta1 CA, OU = Operating
        Subject Public Key Info:
...

I'd very much appreciate if somebody could provide a hint into which direction I should poke more. Thanks!

Edit: turned out the indentation of tls_config and authorization was wrong! those two fields belong to http_sd_configs. Its working now :partying_face:

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.