Improve support for observing multiple clusters
Currently, the charts have limited support for observing multiple independent Pulsar clusters, based on the cluster label. My suggestion is that the charts be improved to support multiple BookKeeper clusters and ZooKeeper clusters, and to do it in a standards-based way.
Specifically my ask is that job not be treated as a constant (bookie, etc.) but as representing a unique cluster. For example, imagine that I have two separate BK clusters. There'd be two Prometheus jobs configured, one for each BK cluster. The name of the job would be user-defined and likely represents the name of the BK cluster (e.g. bookkeeperset/cluster-1). For simple setups, the job name would likely be bookie as before.
This approach seems to be standards-based. From Prometheus docs (reference):
In Prometheus terms, an endpoint you can scrape is called an instance, usually corresponding to a single process. A collection of instances with the same purpose, a process replicated for scalability or reliability for example, is called a job.
I observe that the Prometheus UI and the Prometheus operator seem to assume this sort of setup. For example, the UI paints a given job in red if any of the instances of that job are down. This would make no sense if the job consisted of instances across numerous independent clusters. Meanwhile the Prometheus operator generates job names based on the ServiceMonitor name (which would be service-specific).
In some cases it may be necessary to provide a 'job selector' widget on the dashboard, which would set the job variable. Most of the queries seem to use $job already as a filter, which is good.