Multiple MetricsInstances for the same tenant are not scraped

[Fantaztig]

Grafana Agent Version: 0.23.0 Using Grafana Agent Operator v0.23.0 Kubernetes Version: 1.22

Hi! We have a multi-tenant kubernetes cluster and in general every tenant/namespace has their own MetricsInstance sending to their specific Mimir OrgId. We give tenants a baseline of metrics via the additionalScrapeConfigs parameter. For internal monitoring purposes we have configured multiple namespaces to send their metrics to the same Mimir OrgId.

Now we noticed that only metrics of one of the namespaces are present in Mimir, checking the grafana agent scrape config we can see that the config blocks for all namespaces are there, they simply seem to be ignored. When we change the tenant in one of the "ignored" MetricsInstances to something different the metrics are sent fine.

We expected this to work as the same setup is working fine for logs. Any tips or hints on how we can configure this in a generic way? Currently we fall back to using a single MetricsInstance for multiple namespaces as a workaround.

PS: So far we only tested this with scrape configs defined in the additionalScrapeConfigs, not with ServiceMonitors etc.

[rfratto]

Can you send us the generated agent config? (Feel free to blank out any sensitive information before sending it to us)

[Fantaztig]

Sure thing! I hope the formatting didn't get corrupted, I removed all the configs besides the two that are clashing. scrape_config.txt

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed in 7 days if there is no new activity. Thank you for your contributions!

[Fantaztig]

Still relevant