karpenter
karpenter copied to clipboard
kubernetes-sigs
Add NodePoolGroupLimit CRD for limits that span NodePools
Description
What problem are you trying to solve?
Today, limits can only be specified on individual NodePools. While this is fine for simple situations, it is insufficient when multiple NodePools comprise a logical grouping of compute that should share a limit. This happens most often when there are important variations in a NodePool's properties beyond its requirements that mandate the use of multiple NodePools, but when they are otherwise related in some way relevant to limits (e.g. same department, team, application, budget line-item).
For example, an organization might group limits by team. A team might require nodes labelled in two distinct ways, necessitating the use of two NodePools. Splitting the team's limit in half for each NodePool might not be sufficient if the balance of nodes between the NodePools varies over time.
I propose a NodePoolGroupLimit CRD (or a similar appropriate name) that would allow a defined limit to apply to NodePools chosen by a selector. If multiple NodePoolGroupLimit objects select the same NodePool, the most prohibitive limit should take precedence.
It might look something like this:
apiVersion: karpenter.sh/v1
kind: NodePoolGroupLimit
metadata:
name: frontend-bingo
labels:
team: frontend
service: bingo
spec:
selector:
# label selector for NodePool labels
team: frontend
service: bingo
limits:
cpu: "100"
memory: 128Gi
How important is this feature to you?
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
This issue is currently awaiting triage.
If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
Based on your request, it seems like it's common for teams to have multiple NodePools, where there's overarching org-wide constraints across the cluster. It seems like you also don't want a necessarily global limit across the cluster too, you want something in between.
Are you willing to open up an RFC to talk about your proposed solution and alternatives to the solution that you've explored?
Based on your request, it seems like it's common for teams to have multiple NodePools, where there's overarching org-wide constraints across the cluster.
Not quite - in my case there are team constraints that need to be applied across multiple NodePools belonging to each team. It'd be insufficient for there to be one NodePool per team, as teams require several different configurations that cannot be expressed using a single NodePool.
It seems like you also don't want a necessarily global limit across the cluster too, you want something in between.
Correct
Are you willing to open up an RFC to talk about your proposed solution and alternatives to the solution that you've explored?
Yes
@njtran as it is currently very unlikely that a single node pool could represent even a basic group of compute (even support for AMD64 & ARM64 or support for on-demand & spot can't be handled by a single node pool without significant effort), I'd suggest that this is required for almost all real world scenarios where limits are required.
If done correctly (support {} selector) this approach could also work for #745.
Does this relate to https://github.com/kubernetes-sigs/karpenter/issues/1703?
@sftim I think it's related. Borrowing solves the requirement for limiting based on a node type where multiple node pools are required to implement the node type (e.g. multi-arch without a webhook). But at least in our use case we want a cluster limit for all node types where borrowing isn't relevant.
@jonathan-innis: This request has been marked as needing help from a contributor.
Guidelines
Please ensure that the issue body includes answers to the following questions:
- Why are we solving this issue?
- To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
- Does this issue have zero to low barrier of entry?
- How can the assignee reach out to you for help?
For more details on the requirements of such an issue, please see here and ensure that they are met.
If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.
In response to this:
/help
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.