For tenant prefix, prefer the prometheus tenant id

[matthewjstanford]

trafficstars

In our environment we'd like to use a single deployment of cortex-tenant to manage input from multiple prometheus instances. Each prometheus instance is already setting a tenant id on the request, but that tenant id is lost when it hits cortex-tenant.

It would be nice if we could preserve the source tenant id, and use it as the the tenant prefix when cortex-tenant writes it to the backend.

I'm thinking something along these lines:

tenant:
  prefix: default-prefix
  prefix_prefer_source: true

Example:

Prom-A sends metrics to cortex-tenant with X-Scope-OrgID: Prom-A, and contains metrics with namespace labels that will be translated to a tenant: namespace: app1. Assuming we're using namespace as the tenant label in cortex-tenant. These metrics would be mapped to a new tenant called Prom-A-app1.

Prom-B also sends to the same cortex-tenant instance with X-Scope-OrgID: Prom-B and metric label namespace: app1. This would be translated to Prom-B-app1.

A third prometheus that has no tenant set would be mapped to default-prefix-app1.

And so on.

The reason this is preferred over simply having multiple cortex-tenant deployments is that it keeps the infra as simple as possible. We're decoupling the cortex-tenant deployments from the prometheus deployments.

Thoughts? I've made this change locally and it works as expected. Would be happy to create a PR if this is something that would be useful for other folks.