Dashboards for distributed deployments require undocumented config

[yvrhdn]

The dashboards in tempo-mixin use a cluster and job matcher: https://github.com/grafana/tempo/blob/29c6e25f5c0f9750b16fca673d437a490a6e3069/operations/tempo-mixin/dashboard-utils.libsonnet#L31-L32

This requires a cluster and job label to be present in the time series, which is not documented anywhere.

We should either:

• document what is expected to use this dashboard
• adapt the dashboard to not require this

Ideally, we also update the tk and helm examples to use these dashboards out of the box.

[yvrhdn]

fyi, adding an external label worked for me.

Using the Grafana Agent:

prometheus:
  global:
    external_labels:
      cluster: k3d-local-dev
  # ...

[B3DTech]

This seems like it should have been taken care of by now, but I'm still experiencing the issue of the missing cluster label. Any ideas?

[yvrhdn]

This should be fixed by https://github.com/grafana/tempo/pull/1913