monitoring icon indicating copy to clipboard operation
monitoring copied to clipboard

`StalePersistentVolumeClaim` duplicate time series in Victoria Metrics

Open dhess opened this issue 9 months ago • 0 comments

Hi,

The StalePersistentVolumeClaim query here breaks Victoria Metrics in some configurations:

https://github.com/openebs/monitoring/blob/161b5af7f7525c223b9165a13db2d6b667d08aad/deploy/charts/rules/volume/volume-rules.json#L12

422: error when executing query="kube_persistentvolumeclaim_info unless (kube_persistentvolumeclaim_info * on(persistentvolumeclaim) group_right kube_pod_spec_volumes_persistentvolumeclaims_info) == 1" on the time range (start=1700931915000, end=1700932215000, step=15000): cannot execute "kube_persistentvolumeclaim_info unless ((kube_persistentvolumeclaim_info * on(persistentvolumeclaim) group_right() kube_pod_spec_volumes_persistentvolumeclaims_info) == 1)": cannot execute "(kube_persistentvolumeclaim_info{persistentvolumeclaim=~\"audit-vault-0|audit-vault-1|audit-vault-2|... duplicate time series on the left side of `* on(persistentvolumeclaim) group_right()`: ...

In our particular case, this happens on any ReadWriteMany PVC that occurs in more than one pod in the same namespace.

dhess avatar Nov 25 '23 19:11 dhess