Cortex completely ignores Thanos labels

[SuperMatt]

Describe the bug Cortex completely ignores the Thanos labels after migrating from Thanos to Cortex.

In our environment, we run Prometheus instances on a per cluster basis, and use Thanos for the eventual combining of data from each of the region. We use external labels to keep track of which cluster the data is being scraped from. E.g., a Prometheus instance running in us-east-1 would have a region external label with that information. The targets being scraped are not always aware of the region they're in, so we add external labels at the Prometheus level, and this has worked well for us for a long time.

We now wish to migrate our data to Cortex, but are being held back by the fact that the external labels - clearly present in the meta.json files - are not read by Cortex in the same way that they are by Thanos. We want to move to Cortex for the extra stability it would offer us, especially as all the components are horizontally scalable, but the labels being ignored means that we're losing a lot of useful information that teams within the company heavily rely on.

To Reproduce Steps to reproduce the behavior:

1. Copy Thanos data from Thanos s3 bucket to cortex s3 bucket
2. Use thanoscovert to convert the data
3. Query data that should have external labels applied, note that they are missing.

Expected behavior I would expect to see the previous Thanos external labels on the newly migrated and converted Cortex metrics.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: Helm

Storage Engine

• [x] Blocks
• [ ] Chunks

Additional Context We're not necessarily looking for a bug fix here. Instead I'd like to know why the labels are ignored, and maybe what the expected migration path should look like.

[alanprot]

I believe this was just not implemented yet? PRs are welcome! :D

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[yeya24]

I think this is still valid.