pipeline support for metrics

[oscni]

Would be very useful with pipeline support for metrics. Specifically the tenant function. https://grafana.com/docs/loki/latest/clients/promtail/stages/tenant/

[tpaschalis]

Hey there 👋 thanks for the suggestion! We're exploring adding a more pipeline-like functionality into the Agent, so stay tuned for more news in the future!

About the specific tenant functionality, we don't currently support something as fancy as Loki's solution.

What you could do (at least for now) is to use the headers field under remote_write, to configure an extra header along your metrics. The header value could also be more dynamic by using an env var and passing the -config.expand-env CLI flag.

remote_write:
- url: <remote-url>
  headers:
    X-Scope-OrgID: "my-custom-value"

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed in 7 days if there is no new activity. Thank you for your contributions!

[akselleirv]

Hello @tpaschalis, I would be interested in this feature.

Would it require a lot of effort to implement this functionality? Might be something I could help contribute.

[tpaschalis]

Hey @akselleirv! Thanks for expressing interest in improving Grafana Agent!

The issue here is that Prometheus' remote_write protocol doesn't have the notion of multi-tenancy itself. So, different backends (Mimir, Thanos, Cortex, Timescale etc) use different methods on their own, either via HTTP headers, or 'special' labels. Furthermore, when sending to remote_write Prometheus batches metrics together using its queue manager, so you don't always have fine-grained control in how metrics will get sent to disambiguate their HTTP headers.

So many times the header might be injected by a reverse proxy, or even do the transformation from labels to X-Scope-OrgID header with a different solution like cortex-tenant.

All in all, the effort might depend on the way that we choose to implement this, so if you're still interested you might open a new issue with a proposed solution, so we can kickstart a discussion about how feasible it is. If you're interested in Flow our newest experimental system for building telemetry pipelines, such functionality might tie well into being a standalone Flow component.

[oscni]

Yeah it's true that it's not built in the remote_write protocol and I could create a new issue with just requesting that it would be nice to have headers of your choosing dynamically set from some label. That would be fine for us since we just want to separate kubernetes namespaces into different tenants. But the promtail pipeline is more flexible than that and it I thought it would be good to harmonize the configuration between logs and metrics.

[tpaschalis]

it would be nice to have headers of your choosing dynamically set from some label.

I agree with that. That might be easier when building a pipeline with Flow, but I think that the current Prometheus remote_write code that we're reusing will opportunistically batch metrics and send the RW request as they come, and there's no way to have a more fine-grained manipulation to be able to batch and inject headers depending on a label.

I've opened issue grafana/alloy#521 for implementing a Flow component that achieves this, but we'd have to dig a little deeper to see if it's possible at all from the Agent side when using the YAML configuration, or if we must start a conversation with upstream Prometheus about it.