Allow relabel while compacting blocks

[vibgy]

Is your proposal related to a problem?

Some labels have super high cardinality but are relevant for a short time.

For example - pod_name is useful compared to IP address (instance label) if you are looking at relatively recent metrics, but after these pods get deleted, pod_name is not relevant.

We would like to be able to drop labels while compacting blocks.

Describe the solution you'd like

I would like to be able to drop some labels while compacting blocks (in Thanos compactor) (e.g. Drop pod_name label while creating a 2d block out of multiple 8h blocks)

Describe alternatives you've considered

We have tried to do it manually using Thanos bucket rewrite, but it is error prone if the compactor is running at the same time. Moreover, it’s not efficient because a new block has to be created and the old one has to be deleted.

Additional context

There is a precedence for a client provided mergeFunc (here) to the tsdb.LeveledCompactor that manipulates SeriesChunk sets. Along the same lines, we could extend the tsdb compactor to allow for providing a relabelFunc. This would allow a client to modify the Symbols and Labels before writing the compacted block. For example - This would allow thanos compactor to pass in a relableFunc while compacting the appropriate block with user provided relabelConfig.

If this proposal makes sense to the Thanos community, then I can create a feature request in Prometheus repo and can also create PRs.

If you made it this far, thank you for reading. Would love you inputs and ideas.

[yeya24]

e.g. Drop pod_name label while creating a 2d block out of multiple 8h blocks

This makes more sense to do some aggregation like recording rule instead of dropping labels directly.

There is a precedence for a client provided mergeFunc (here) to the tsdb.LeveledCompactor that manipulates SeriesChunk sets. Along the same lines, we could extend the tsdb compactor to allow for providing a relabelFunc. This would allow a client to modify the Symbols and Labels before writing the compacted block. For example - This would allow thanos compactor to pass in a relableFunc while compacting the appropriate block with user provided relabelConfig.

Maybe you can raise this to the Prometheus repo and see how we can extend the compactor to fullfil your requirements.

[bwplotka]

As mentioned offline, I love the idea and fact we do that during compaction here, but I believe we have a room in providing better user experience and somehow marry the relabel with recording rule and removal of unnecessary series. Ideally we can have such request go around Prometheus ecosystem for more opinions (:

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale[bot]]

Closing for now as promised, let us know if you need this to be reopened! 🤗