Return labels as part of the returned datapoints

[weeco]

Hey, I'd like to access the labels as part of the returned datapoints. Let me describe my usecase in more detail:

Similiar like I'd do with Prometheus I'm collecting several metrics for Kafka consumer groups. Each consumer group may consume one or more Kafka topics. The consumer group may have a delay/lag which is different for each topic. I'm scraping this lag for each <consumerGroup, topic> tuple and later on I'd like to render a graph of the lag for each of this tuple.

This would currently require me to know the labels in advance since they are not returned in storage.Select() (just the value and timestamp are returned as part of the DataPoint structure). If I had an option to also return the labels like this:

storage.Select("consumergroup_topic_lag", []tstorage.Label{"consumer_group": "foo"}, start. end, tstorage.WithLabels())

that would be very helpful. I would be able to group the returned datapoints by a certain label key and therefore further process my results.

[nakabonne]

@weeco Hey, Unfortunately, it currently only returns data points with an exact match of the label combination, which means, let's say you run storage.Select("consumergroup_topic_lag", []tstorage.Label{"consumer_group": "foo"}) then:

• consumergroup_topic_lag {"consumer_group": "foo"} will be returned
• consumergroup_topic_lag {} will not be returned
• consumergroup_topic_lag {"consumer_group": "foo", "another_label": "bar"} will not be returned

Because of that, all returned data points have the same labels.

For my original use case that works fine, but obviously not versatile. I'm wondering if we had better completely change the labeling mechanism.

If you can expect all possible labels, you are able to workaround by querying it with all possible labels (definitely not effective though).

[weeco]

Oh I get it, your example is very clear. I actually expected that the labels do not need to match exactly. Should I create a separate issue for this as some sort of feature request or is this something you don't can / want to put further time into anyways?

If you can expect all possible labels, you are able to workaround by querying it with all possible labels (definitely not effective though).

Since the labels need to match exactly this doesn't make any sense I think. The response would only contain datapoints that have the exact same labels as far as I got. Thus partial label matches would be a requirement I think there are many usecases that could benefit from this feature.

[nakabonne]

Should I create a separate issue for this as some sort of feature request or is this something you don't can / want to put further time into anyways?

It would be great if you could open another issue because I believe that is worth considering.