kube-state-metrics icon indicating copy to clipboard operation
kube-state-metrics copied to clipboard
verified publisher icon kubernetes

fix: fix the issue that duplicated CR metrics are generated when CRD is deleted/applied multiple times

Open CatherineF-dev opened this issue 2 years ago • 9 comments

What this PR does / why we need it: https://github.com/kubernetes/kube-state-metrics/issues/2223

# It returns same metrics data many times when CRD is deleted/applied multiple times
 curl localhost:8089/metrics
# HELP cr_creationtimestamp
# TYPE cr_creationtimestamp gauge
cr_creationtimestamp{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 1.701828773e+09
# HELP cr_resourceversion
# TYPE cr_resourceversion gauge
cr_resourceversion{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 909919
# HELP cr_creationtimestamp
# TYPE cr_creationtimestamp gauge
cr_creationtimestamp{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 1.701828773e+09
# HELP cr_resourceversion
# TYPE cr_resourceversion gauge
cr_resourceversion{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 909919

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Mitigation for https://github.com/kubernetes/kube-state-metrics/issues/2223

CatherineF-dev avatar Dec 06 '23 01:12 CatherineF-dev

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: CatherineF-dev Once this PR has been reviewed and has the lgtm label, please assign mrueg for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot avatar Dec 06 '23 01:12 k8s-ci-robot

I don't think this will fix the inherent issue. It seems that the mechanism detecting the deletion of CRD matched by the ksm config is not working properly. Normally when that happens, the internal store should be deleted alongside its informer because we don't want to expose metrics about this resource anymore.

dgrisonnet avatar Dec 06 '23 12:12 dgrisonnet

To be clearer, the scenario where a specific resource appears twice in ksm code should never happen, so we should understand why it happened and fix it instead of enforcing uniqueness.

dgrisonnet avatar Dec 06 '23 12:12 dgrisonnet

got it, agree. Will debug again to find root cause.

CatherineF-dev avatar Dec 11 '23 19:12 CatherineF-dev

It should be related to https://github.com/kubernetes/kube-state-metrics/pull/1851 where adding PollForCacheUpdates function.

CatherineF-dev avatar Dec 13 '23 11:12 CatherineF-dev

It deletes removed CR here https://github.com/rexagod/kube-state-metrics/blob/main/internal/discovery/discovery.go#L42C24-L116, while didn't delete removed CR here https://github.com/rexagod/kube-state-metrics/blob/25a1d8da057cf761d614c59a52785335d34082d1/internal/discovery/discovery.go#L232.

How about creating a new issue and merging this PR as a mitigation? I think it needs some changes around https://github.com/kubernetes/kube-state-metrics/pull/1851.

Or we only want the fix PR instead of mitigation.

cc @dgrisonnet

CatherineF-dev avatar Dec 13 '23 12:12 CatherineF-dev

I second https://github.com/kubernetes/kube-state-metrics/pull/2257#issuecomment-1842797358, and as for https://github.com/kubernetes/kube-state-metrics/pull/2257#issuecomment-1853814510, I'd prefer the changes in this PR (instead of mitigating the issue, we want to identify the root cause and fix that since duplicacy shouldn't ideally be exhibited at all).

rexagod avatar Jan 11 '24 17:01 rexagod

any updates here? It leads to losing the metrics, which is not good.

korjek avatar Mar 05 '24 12:03 korjek

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle stale
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jun 03 '24 12:06 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle rotten
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Jul 03 '24 13:07 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Reopen this PR with /reopen
  • Mark this PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-triage-robot avatar Aug 02 '24 13:08 k8s-triage-robot

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Reopen this PR with /reopen
  • Mark this PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot avatar Aug 02 '24 13:08 k8s-ci-robot

/reopen

korjek avatar Aug 02 '24 14:08 korjek

@korjek: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot avatar Aug 02 '24 14:08 k8s-ci-robot