External labels not applied to alerts

[lennartack]

I have configured Prometheus with external_labels. If I understand correctly the Thanos sidecar should add these labels to metrics and rules when they are proxied to the querier. This works on my setup for metrics. However, it seems external labels are not applied to alerts. The /api/v1/alerts endpoint on the Thanos querier gives the alerts without external labels.

[GiedriusS]

Thanos Query also deduplicates alerts so I think that's why you don't see them. Are they visible if you removed replica labels from Query?

[lennartack]

@GiedriusS The only replica label I have configured is replica. The external label I'm looking for is called cluster. I don't think it is deduplication as the cluster label is applied to metrics, just not to alert instances.

[lennartack]

I can also see the labels in the alert definitions at /api/v1/rules?type=alert. But not the instances at /api/v1/alerts. The latter is quite important for us for viewing alert instances in Grafana per cluster.

[viktordaniel]

:+1: +1 on it, we have same problem.