[Bug] Inconsistency between documentation and real behavior about kyverno_policy_results metric.

[benoitpuertolas]

Page link

https://kyverno.io/docs/monitoring/policy-results-info/

Description

On the documentation, the policy and rule execution metric is called kyverno_policy_results_total and is described as a Counter.

On my 1.12.1 Kyverno installation, the /metrics endpoint is giving a kyverno_policy_results metric as an histogram (kyverno_policy_results_bucket, kyverno_policy_results_sum and kyverno_policy_results_count). I don't understand what this metrics means and this results in a metric with a cardinality a lot higher.

/metrics partial output:

# HELP kyverno_policy_results can be used to track the results associated with the policies applied in the user's cluster, at the level from rule to policy to admission requests
# TYPE kyverno_policy_results histogram
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="0.005"} 0
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="0.01"} 0
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="0.025"} 0
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="0.05"} 0
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="0.1"} 0
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="0.25"} 0
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="0.5"} 0
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="1"} 12
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="2.5"} 12
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="5"} 12
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="10"} 12
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="15"} 12
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="20"} 12
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="25"} 12
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="30"} 12
kyverno_policy_results_bucket{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate",le="+Inf"} 12
kyverno_policy_results_sum{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate"} 12
kyverno_policy_results_count{otel_scope_name="kyverno",otel_scope_version="",policy_background_mode="true",policy_name="add-safe-to-evict",policy_namespace="-",policy_type="cluster",policy_validation_mode="audit",resource_namespace="cert-manager",rule_execution_cause="admission_request",rule_result="skip",rule_type="mutate"} 12

Expected behavior

I don't know if this is a regression in kyverno itself or a undocumented change. I would like the kyverno behavior about policy and rule execution metrics to reflect the documention.

Slack discussion

No response