dra-example-driver icon indicating copy to clipboard operation
dra-example-driver copied to clipboard

Delete resourceclaims on job completion or termination

Open asm582 opened this issue 2 years ago • 8 comments

When batch/v1 job requesting a GPU claim through resourceclaim is completed, the underlying resourceclaim is still in allocated, reserved state. This will lead to resource wastage. I think we should delete the claim if the object consuming the resourceclaim completes or terminates

asm582 avatar Dec 04 '23 19:12 asm582

This has been discussed in the past but in any case has nothing to do with the specifics of NVIDIAs DRA driver for GPUs (this is an issue for upstream Kubernetes where I believe a similar issue has already been created).

klueska avatar Dec 04 '23 19:12 klueska

To be clear tough — the claim itself cannot (and should) not be deleted. Only the resources allocated to the claim should be freed but the claim itself should remain.

klueska avatar Dec 04 '23 19:12 klueska

Can you share more light why claim should not be deleted?

asm582 avatar Dec 04 '23 20:12 asm582

Just as with any other API server object -- if the user creates the claim, it should be the user that also deletes it. Claims created as part of a ClaimTemplate will be cleaned up by the system because the claims were created by the system. But claims created by the user (e.g. via a kubectl apply) will only be deleted upon request by the user (e.g. via a kubectl delete).

klueska avatar Dec 04 '23 20:12 klueska

Which version of Kubernetes was used? I just tested with 1.28 and our latest DRA driver. Just like I remembered - this has been fixed some time ago - 1.27 or 1.28, the deallocatoin part at least. The Claims created manually stay, but they are no longer allocated, and Claims generated even get deleted.

byako avatar Dec 04 '23 22:12 byako

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Mar 03 '24 22:03 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Apr 02 '24 22:04 k8s-triage-robot

@klueska is this the issue you had in mind? https://github.com/kubernetes/kubernetes/issues/120756

kannon92 avatar Apr 18 '24 01:04 kannon92

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-triage-robot avatar May 18 '24 02:05 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot avatar May 18 '24 02:05 k8s-ci-robot