kubebuilder
kubebuilder copied to clipboard
kubernetes-sigs
Custom resource metrics
What do you want to happen?
Prior to the Operator Framework's Operator SDK re-architecting itself as a Kubebuilder plugin, it contained libraries (since deleted in operator-framework/operator-sdk#3249) for generating and serving custom resource metrics (implemented using libraries from kube-state-metrics) and scaffolded operators to invoke said libraries on startup. As a result, operators built using the Operator SDK not only exposed the standard controller-runtime metrics, but also simple metrics about the existence of custom resources managed by the controller (i.e. <KIND>_info{<KIND>="<NAME>", namespace="<NAMESPACE>"} 1).
It'd be nice to provide these metrics for free via Kubebuilder's scaffolding or at least provide an easy way for operator developers to opt-in.
If possible, it'd also be nice to provide an easy, pluggable way for operator developers to declaratively instrument their operators with custom (adjective) custom resource metrics without having to do so in Reconcile (as documented by https://book.kubebuilder.io/reference/metrics.html#publishing-additional-metrics). For example, if a custom resource has a simple string field in its status subresource, it'd be nice to provide an API that allows operator developers to provide a metric name suffix, label name, and function for deriving label values from custom resources so that the controller generates and serves these metrics separately from Reconcile - preventing developers from having to instrument Reconcile wherever they update the status subresource's string field.
I wasn't sure whether to open this issue against kubebuilder or controller-runtime. I'm more than happy to move it to controller-runtime if desired. I also considered opening this issue against https://github.com/operator-framework/operator-lib since it's maintained as part of the Operator Framework. Alternatively, I could imagine the simple "info" metrics for custom resources being a feature request against kube-state-metrics. What do you all think?
Thanks!
Extra Labels
No response
RE my kube-state-metrics comment at the end, it looks like there have been some feature requests over there. The older of these two was closed because the maintainers felt that such metrics were the responsibility of operators. The other is still open and it looks like implementation is underway
https://github.com/kubernetes/kube-state-metrics/issues/457 https://github.com/kubernetes/kube-state-metrics/issues/1210
EDIT
It looks like the second of these two is focusing on custom resource definitions, not custom resources
@robbie-demuth Sounds interesting! Could you please describe some use cases on how you would use these standard controller runtime and metrics about existence of CRs?
The simple "info" metric without additional labels could be used to count custom resources by namespace, cluster, etc. Making additional labels or metrics pluggable, however, really opens up the door to anything
Hi @robbie-demuth,
Thank you to raise this issue.
Note that it was desirable in the past: https://github.com/kubernetes-sigs/kubebuilder/issues/1068 I think that would be fit very well in a plugin to allow opt-in. See: https://book.kubebuilder.io/plugins/plugins.html WDYT?
Please, feel free to contribute to this implementation. Your collab is very welcome.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
Maybe worth to note that kube-state-metrics now supports custom resource definitions: https://github.com/kubernetes/kube-state-metrics/pull/1710
Although it does not yet work for _info metrics: https://github.com/kubernetes/kube-state-metrics/issues/1755
@Kavinjsir,
This one is related to the metrics so you might like to check it out. I think we need just to see how we could add the kube-state-metrics to kubebuilder.
Would be a great thing to have and at the Cluster API project would be interested in it too
- xref: https://github.com/kubernetes-sigs/cluster-api/issues/7158
I was about to ask the question in a new issue at kube-state-metrics if that is a reasonable way forward.
Happy to help working on it!
This issue has not been updated in over 1 year, and should be re-triaged.
You can:
- Confirm that this issue is still relevant with
/triage accepted(org members only) - Close this issue with
/close
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted
kube-state-metrics now provides a declarative way to define custom resource state metrics, which our organization uses with great success, so I'm going to go ahead and close this
https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md