Allow-list Kubernetes annotations to add as labels

[brancz]

Same as we already attach labels, it would be nice to be able to choose annotations to add as labels. Since annotations are often generously used in Kubernetes it probably makes sense to allow-list them instead of applying all by default.

[nabokihms]

@brancz With the current labeling approach, default labels may be overwritten with the Kubernetes labels, and if we add annotations, also with annotations.

For labels and annotations, wouldn't it be better to prefix labels the way Prometheus did for targets, e.g., __metadata_kubernetes_label_, __metadata_kubernetes_annotation_? Users will be able to modify them as they wish using relabeling rules.

To keep backward compatibility, for some time, it is possible to store labels with both prefixed and non-prefixed keys.

[brancz]

I think my preferred solution would be to have everything be as it is in Prometheus, where users can create their own relabel configs to relabel exactly the labels they want onto targets.

We can maintain today’s behavior by having a default config that is applied when people don’t provide any config themselves.

[brancz]

Would be extremely happy about contributions for this!