kube-state-metrics icon indicating copy to clipboard operation
kube-state-metrics copied to clipboard

feat: introduce deletion timestamp metric for multiple resources

Open IgorIgnatevBolt opened this issue 6 months ago • 3 comments

What this PR does / why we need it:

Some resources can be blocked by deletion from finalizers. To catch this and expose it to metrics, we can use the deletion timestamp metadata field. Introduce a deletion_timestamp metric for the next resources:

  • deployment kube_deployment_deletion_timestamp
  • statefulset kube_statefulset_deletion_timestamp
  • daemonset kube_daemonset_deletion_timestamp
  • service kube_service_deletion_timestamp
  • poddisruptionbudget kube_poddisruptionbudget_deletion_timestamp

Also formatting tables in docs

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #

IgorIgnatevBolt avatar Jun 02 '25 11:06 IgorIgnatevBolt

All commits were squashed into one.

IgorIgnatevBolt avatar Jun 03 '25 16:06 IgorIgnatevBolt

Hi, could you share more insights on use cases after these metrics are added?

Is it used for monitoring Kubernetes resources that are stuck in a terminating state?

CatherineF-dev avatar Jun 20 '25 00:06 CatherineF-dev

@CatherineF-dev Hi, yes, if the resource deletion process is stuck for some reason or blocked by the finalizer, deletiontimestamp metric can help to detect such a case and raise an alert for investigation.

IgorIgnatevBolt avatar Jun 20 '25 05:06 IgorIgnatevBolt

/assign

richabanker avatar Jun 26 '25 16:06 richabanker

@IgorIgnatevBolt How will we know which resource should be deleted?

CatherineF-dev avatar Jul 11 '25 12:07 CatherineF-dev

@IgorIgnatevBolt How will we know which resource should be deleted?

Maybe I misunderstood the question, but this PR is exactly about detection for such resources that were nominated by the controller manager for deletion but not deleted for some reason, eq blocked by finalizers

The controller managing that finalizer notices the update to the object setting the metadata.deletionTimestamp, indicating deletion of the object has been requested.

IgorIgnatevBolt avatar Jul 11 '25 12:07 IgorIgnatevBolt

Hi @CatherineF-dev, do you need any more information about PR or anything else that can help you move forward?

IgorIgnatevBolt avatar Jul 30 '25 07:07 IgorIgnatevBolt

/assign @CatherineF-dev /triage accepted

dgrisonnet avatar Aug 07 '25 16:08 dgrisonnet

rebased, conflicts solved

IgorIgnatevBolt avatar Aug 15 '25 06:08 IgorIgnatevBolt

fixed typo in the depl test

IgorIgnatevBolt avatar Aug 18 '25 05:08 IgorIgnatevBolt

/hold

for @CatherineF-dev to further comment. Renamed the title to include all additional resources in there.

/lgtm

mrueg avatar Aug 20 '25 17:08 mrueg

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: IgorIgnatevBolt, mrueg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot avatar Aug 20 '25 17:08 k8s-ci-robot

/hold cancel

mrueg avatar Aug 25 '25 05:08 mrueg