kubernetes
Custom resource metrics - behaviour when currently CRD doesnt exists on cluster
Hello
In our solution we are dynamically install/uninstall CRDs during cluster lifetime based on current configuration. We wanted to use CustomResourceStateMetrics to track this CRs Statuses.
Problem goes out when some CRD is not yet installed or it is already deleted(and in future it may come again or not)
Currently, If we have setup CustomResourceStateMetrics in Helm config for our CR and there is no CRD on cluster, there is a lot of error logs coming from kube-state-metrics saying that it cannot watch this resource.
Is it possible to somehow configure custom metrics to watch only this CRDs which are actually installed on cluster, and if there is no such CRDs then ignore it, without spam of log errors? if it is not, can you advise some recommended solution how to handle this situation?
One solution is to add log verbose level to these spam logs.
Do you know which line is generating these spam logs? You might can find it from error logs.
example error line from kube-state-metrics logs:
E0629 12:34:11.050120 1 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch operator.kyma-project.io/v1alpha1, Kind=BtpOperator: failed to list operator.kyma-project.io/v1alpha1, Kind=BtpOperator: the server could not find the requested resource
i was trying to set errorLogV: 0 on CustomResourceStateMetrics CR and also i was trying to set --v=0 as arg for kube-state-metrics cli but it didnt stop to log errors.
This error is from https://github.com/kubernetes/client-go/blob/master/tools/cache/reflector.go#L535.
However, hard to configure log level for client-go package. If contextual logging is used in client-go, it can be possible.
Example PR on using contextual logging: https://github.com/kubernetes/kubernetes/pull/118071/files
This issue has not been updated in over 1 year, and should be re-triaged.
You can:
- Confirm that this issue is still relevant with
/triage accepted(org members only) - Close this issue with
/close
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted
