Make consolidation configurable

[hgambarian]

Description

What problem are you trying to solve? Currently when the node is underUtilized, or consolidable, the controller cordons the node and drains immediately. But when we have karpenter.sh/do-not-evict: true, the disruption will be blocked.

I want something like:

• If the node is underUtilized - Cordon the node
• When the jobs finish their stuff and completed, the node becomes empty
• When the node is empty, drain,kill the node

How important is this feature to you?

• Huge cost-optimization

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

Does #624's suggestion to use a PreferNoSchedule for consolidation solve this issue combined with #916? With the addition of both of these features we would: 1) Ignore the karpenter.sh/do-not-disrupt annotation when considering candidates and only respect them during the drain operation and 2) Taint the node as PreferNoSchedule when we see that they are underutilized, which should cause pods to be pushed away from the node during the consolidateAfter timeframe?

[jonathan-innis]

/assign jonathan-innis

[hgambarian]

Does #624's suggestion to use a PreferNoSchedule for consolidation solve this issue combined with #916? With the addition of both of these features we would: 1) Ignore the karpenter.sh/do-not-disrupt annotation when considering candidates and only respect them during the drain operation and 2) Taint the node as PreferNoSchedule when we see that they are underutilized, which should cause pods to be pushed away from the node during the consolidateAfter timeframe?

Thanks, looks like this is what I want.

[eden881]

Our team needs exactly that. We use our cluster to run a mix of workload types, some are services which can be evicted normally, and some are processes (such as Argo workflow pods) which should not be interrupted when possible. We annotate the process pods with karpenter.sh/do-not-disrupt and it blocks consolidation as it should, but as the cluster schedules more process pods they land freely on those nodes that should otherwise get disrupted.

If Karpenter could preemptively cordon a node which it wants to disrupt and wait for pods annotated with karpenter.sh/do-not-disrupt to finish, it would be a huge cost saver and a game-changing feature for us.