Consistent colours for pods in Grafana

[tombentley]

Describe the bug

The provided grafana dashboards don't use a consistent colour across the charts for the same zookeeper node or kafka broker. This can make reading and correlating the charts more difficult than it needs to be. See https://github.com/grafana/grafana/issues/6662

To Reproduce Steps to reproduce the behavior:

1. Just look at the default grafana dashboard provided for a ZK cluster with ≥ 2 nodes.

Expected behavior

You'll see zookeeper-n has different colours in the different timeseries charts. So cyan might be zookeeper-0 in one chart but zookeeper-2 on another.

[scholzj]

Triaged on 2.8.2022: This is still an issue in the current dashboards.

[Frawless]

I did some digging around and I think the only thing we can do is to update prometheus queries to return one value per pod name. I was able to reach a sufficient results with this:

sum(zookeeper_outstandingrequests{namespace="strimzi-kafka",strimzi_io_cluster="heimdall",strimzi_io_kind="Kafka",kubernetes_pod_name=~"heimdall-.*"}) by (pod)

Only Y-axis has different scale, but graph values are the same with this query