kube-state-metrics
kube-state-metrics copied to clipboard
kubernetes
do not create annotation and label metrics if none are allowed
What would you like to be added: Currently kube-state-metrics creates object_annotations and object_labels metrics for all objects even if the objects have no configured allow list.
Why is this needed: This creates many metrics which contain no useful information, especially for pods and replicasets. For example:
kube_pod_annotations{instance="...:8080", job="monitoring/kube-state-metrics", namespace="namespace", pod="podname-bc94dfc78-fcf4j", uid="a482250c-6f6e-4533-a596-3eff484d8969"}
Describe the solution you'd like If the annotation or label has no configured allowed values (--metric-annotations-allowlist, --metric-labels-allowlist) no object_annotations or object_labels metrics should be created.
I'm not sure about removing the default behaviour TBH. I don't see what we stand to gain if such a check is implemented? Is the effort worth the outcome?
I do think so, the annotations provide no value and there are lot of them. You have two for every single pod and replicaset in the cluster. In our clusters this is in the top5 metrics kube-state-metrics produces and it produces a lot. In the grand scheme of things its not super significant but it is also easy to fix.
As for it being breaking, technically true, but you are requesting the metric to not be produced by not allowing any annotations or label metrics. I doubt someone would then rely on an empty metric being produced. The expectation is, if I do not allow label and annotation metrics, I get no annotation and label metrics.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
I tend to agree that such metrics often provide little value and at the cost of increasing cardinality. The backwards incompatible change is a concern though, and I am not sure if it's sufficient to just add a entry in the changelog.
The backwards incompatible change is a concern though, and I am not sure if it's sufficient to just add a entry in the changelog.
I wouldn't be too concerned about that personally. Although it is true that this might be a breaking change, I would consider doing it for 3 reasons:
- The metrics are all experimental so they don't have any guarantees
- Users shouldn't have anything depending on a metric that does nothing and if they do, it is in their best interest to change what they are doing
- It would significantly reduce the memory usage of the default setup of ksm
Circling back to a recent discussion I had offline with @mrueg, it would generally be very beneficial for ksm resource usage to select what we are actually storing. For example if a metric is useless in some scenarios it would be better to just not generate and expose it at all. If a label is empty let's just not store and expose it. This is a bigger effort that should be tackled in a major release since it might have some effect on stable metrics, but for annotations and labels I think it should be fairly safe to do in a minor.
Sounds good to me, happy to have labels and annotations not exposed by default. I think a large majority of users would benefit from this change anyway.
@juliantaylor do you want to take a stab at implementing this new feature? I am happy to give some pointers if needed.
I created PR (https://github.com/kubernetes/kube-state-metrics/pull/2145) to implement the feature.
When --metric-annotations-allowlist and --metric-labels-allowlist are not configured, kube_*_annotations and kube_*_labels will not be provided by kube-state-metrics.
