scylla-operator icon indicating copy to clipboard operation
scylla-operator copied to clipboard

Prometheus metrics relabel is not up to date

Open amnonh opened this issue 1 year ago • 6 comments

What happened?

Scylla Enterprise versions 2023.1.x and higher do not show shard-level metrics in the dashboards.

What did you expect to happen?

NA

How can we reproduce it (as minimally and precisely as possible)?

NA

Scylla Operator version

NA

Kubernetes platform name and version

```console $ kubectl version # paste output here ```

Kubernetes platform info:

Please attach the must-gather archive.

NA

Anything else we need to know?

Look at the metrics relabel config between this line: https://github.com/scylladb/scylla-monitoring/blob/1ab337a505e4f57ec15001480d0898c6d18c8a76/prometheus/prometheus.yml.template#L75 And this line: https://github.com/scylladb/scylla-monitoring/blob/1ab337a505e4f57ec15001480d0898c6d18c8a76/prometheus/prometheus.yml.template#L126

amnonh avatar Jan 04 '24 07:01 amnonh

@amnonh - do we need to update the monitoring in the operator? 4.6 is out, it's a good time to do it?

mykaul avatar Jan 04 '24 11:01 mykaul

I'm removing myself as assigned, assign me if you need more information or if you want to me to do anything

amnonh avatar Jan 04 '24 11:01 amnonh

@amnonh @mykaul's directions were that monitoring team should be updating the dashboard definitions here, that's the reason this was assigned to you.

tnozicka avatar Jan 04 '24 12:01 tnozicka

I've opened the issue around the Prometheus relabel config, which is broken for 2023.1 and higher.

amnonh avatar Jan 04 '24 12:01 amnonh