[proposal] CPUSet injections to container runtime for fine-grained CPU orchestration

[zwzhang0107]

What is your proposal: Setting cgroup/cpuset parameters for containers by pod QoS.

Why is this needed: fine-grained CPU orchestration is supported by koord-scheduler, koordlet should apply the scheduling result to cgroup paramters. See https://github.com/koordinator-sh/koordinator/issues/96 for more details.

Is there a suggested solution, if so, please add it: rules of cpuset:

• LSE/LSR: follow the scheduling result field cpuset from the annotation scheduler.koordinator.sh/alloc-status of pod.
• LS: follow the scheduling result field numaNode or numaSocketfrom the annotation scheduler.koordinator.sh/alloc-status of pod if specified. Or using all cpu share pool if not.
• BE: set as be-cpu-share-pool
• cgroup cpuset should be updated once the alloc-status or share-pool update.

cpuset injections should be implemented as a plugin of runtime hooks; here are two kinds of container injections.

• create: parse cpuset from alloc-status or share-pool, should be injected by the interceptor of plugin.
• update: alloc-status updated or share-pool info update, this will trigger the update of statesinformer, plugin will be executed as a callback. this always happens if LSR/LSE pod rebinds, other share-pool updates due to the pod created/destroyed.

using cri request to modify cpuset is better

for origin k8s pods(without koord-QoS): Guaranteed(without static): treat as LS. Guaranteed(with static): follow the rules from /var/lib/kubelet/cpu_manager_state if present. Or treat as LS if not present. Burstable: treat as LS. Besteffort: treat as BE.

• [x] v0.5 https://github.com/koordinator-sh/koordinator/issues/265
• [ ] v0.6 #346

[zwzhang0107]

/assign zwzhang0107

[zwzhang0107]

/area koordlet

[jasonliu747]

/close

[koordinator-bot[bot]]

@jasonliu747: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.