Clickhouse Keeper prometheus endpoint setup

[noobHappylife]

Is there any example that shows how to setup the clickhouse keeper with proper prometheus metrics endpoint?

I've tried adding the following annotation to the podTemplates:

annotations:
  prometheus.io/port: "7000"
  prometheus.io/scrape: "true"

and apply the grafanadashboard with the following.

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: clickhouse-keeper-dashboard
  namespace: clickhouse-operator-system
spec:
  resyncPeriod: 30s
  instanceSelector:
    matchLabels:
      dashboards: grafana
  allowCrossNamespaceImport: true
  configMapRef:
    name: clickhouse-operator-altinity-clickhouse-operator-dashboards
    key: ClickHouseKeeper_dashboard.json

However, on grafana what I see is all "error". While the clickhouse operator and queries dashboard is working fine.

[discostur]

@noobHappylife i got it working by deploying a separat PodMonitor for the keeper installation:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    app.kubernetes.io/instance: clickhouse-operator
    app.kubernetes.io/name: altinity-clickhouse-operator
  name: clickhouse-keeper-metrics
  namespace: clickhouse-operator
spec:
  podMetricsEndpoints:
  - path: /metrics
    port: metrics
  namespaceSelector:
    matchNames:
    - clickhouse-operator
  selector:
    matchLabels:
      clickhouse-keeper.altinity.com/chk: clickhouse-keeper 

After that i had to improve / fix the provided grafana dashboard because the labels are not up to date, see:

https://github.com/Altinity/clickhouse-operator/pull/1872

[noobHappylife]

Hi, thanks for checking it out. I also managed to have it working with this setup for ClickHouseKeeperInstallation and adding a VMPodscrape (i'm using victoriametrics) that targets this port with the relabeling config.

spec:
  configuration:
    ...
    settings:
      ...
      prometheus/endpoint: /metrics
      prometheus/port: 7000
      prometheus/metrics: true
      prometheus/events: true
      prometheus/asynchronous_metrics: true
  defaults:
    templates:
      podTemplate: clickhouse
      dataVolumeClaimTemplate: clickhouse
  templates:
    podTemplates:
      - name: clickhouse
        metadata:
          labels:
            ...
          name: clickhouse-keeper
          annotations:
            prometheus.io/port: "7000"
            prometheus.io/scrape: "true"
        spec:
          containers:
            - name: clickhouse-keeper
              imagePullPolicy: IfNotPresent
              image: clickhouse/clickhouse-keeper:24.12
              ports:
                - name: chk-metrics
                  containerPort: 7000 

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
  name: clickhouse-keeper-metrics
  labels:
    app.kubernetes.io/part-of: vm-operator
    app.kubernetes.io/instance: vm-pod-scrape
spec:
  selector:
    matchLabels:
      app: clickhouse-keeper
  podMetricsEndpoints:
    - port: chk-metrics # please update the port if changed in the clickhouse keeper config
      relabelConfigs:
        - sourceLabels: [__meta_kubernetes_namespace]
          targetLabel: namespace
        - sourceLabels: [__meta_kubernetes_pod_name]
          targetLabel: pod_name
        - sourceLabels: [__meta_kubernetes_pod_container_name]
          targetLabel: container_name