concourse 6.0 service monitor is not auto discoverd by prometheus operator

[gowrisankar22]

Hello Colleagues,

I have enabled the Prometheus monitor as well as a scrapper. The concourse is running in my default namespace and Prometheus operator is running in monitoring namespace.

Not sure why the service monitor is not auto-discovered by the Prometheus Operator.

@

prometheus:
      enabled: true

      ## IP to listen on to expose Prometheus metrics.
      ##
      bindIp: "0.0.0.0"

      ## Port to listen on to expose Prometheus metrics.
      ##
      bindPort: 9391
      ## If Prometheus operator is used, also create a servicemonitor object
      serviceMonitor:
        enabled: true
        interval: "30s"
        # Namespace the servicemonitor object should be in
        namespace: monitoring

@KYannick can you help with this issue ? Am i missing anythig ?

BRs, Gowrisankar

[KYannick]

@gowrisankar22 Did you configure the "serviceMonitorSelector" and "serviceMonitorNamespaceSelector" of the prometheus-operator in a way the servicemonitor in the monitoring namespace is picked up? Also take a look at the "serviceMonitorSelectorNilUsesHelmValues" parameter, by default this is true, causing prometheus to only select servicemonitors with a specific label (if other values are left to the defaults.)

[gowrisankar22]

@KYannick Thanks a lot for the hint. That did the trick. After setting serviceMonitorSelectorNilUsesHelmValues to false, it worked :)

Now I can see the metrics. Should we design the dashboards for concourse on our own, based on Prometheus metrics or are there any publically available dashboards apart from https://grafana.com/grafana/dashboards/7036 ??

Dashboards from bosh-release are not working :( https://github.com/bosh-prometheus/prometheus-boshrelease/issues/370

[cirocosta]

Hey @gowrisankar22 ,

At the moment we (Concourse) don't really have anything that we support and expose to the community.

We were using Prometheus for a while some time ago, so you could perhaps take a look at some of the dashboards we were using back then: https://github.com/concourse/hush-house/blob/master/deployments/with-creds/metrics/dashboards/concourse/concourse.json

note that it's not only prometheus-based: it also includes Stackdriver.

I think other folks might also have experience tailoring grafana dashboards for Concourse - it might be worth trying to gather some of those that the community has been working on (perhaps try the forum?)

thanks!

[gowrisankar22]

Thanks @cirocosta