Document Pod failures with Out Of Resources error

[AnishShah]

trafficstars

This is a Feature Request

What would you like to be added

Symptoms

A pod is failing with OutOfCpu / OutOfMemory / OutOfPods error. kubectl describe output for such pods:

Name:           pod-566c6c585-jsjhn
Namespace:      namespace
...
Status:         Failed
Reason:         OutOfcpu
Message:        Pod Node didn't have enough resource: cpu, requested: 120, used: 3803, capacity: 3920

Status:         Failed
Reason:         OutOfMemory
Message:        Pod Node didn't have enough resource: memory, requested: 16000000000, used: 31946743808, capacity: 37634150400

Status:         Failed
Reason:         OutOfPods
Message:        Node didn't have enough resource: pods, requested: 1, used: 32, capacity: 32

Root cause

The kube-scheduler is responsible for scheduling and assigning pods to nodes, based on the available resources on each node. However, it is important to note that the kubelet may also have static pods (like kube-proxy) the presence and resource requests of which are not known to the kube-scheduler until they are reported by the kubelet.

In rare cases, it is possible for the following sequence of events to occur during a node bootstrap process:

1. The kubelet starts and registers its node on the kube-apiserver.
2. The kubelet reports the available node resources.
3. The kubelet discovers and runs static pods from the /etc/kubernetes/manifests/ directory.
4. The kube-scheduler assigns pods to the node, the total resource consumption is close to the node's available resources (kube-scheduler doesn't know about the static pod(s) yet).
5. The kubelet accepts and run pods that were scheduled by kube-scheduler.
6. The kubelet reports the updated available node resources, including the static pod(s) resource requests.
7. If the total amount of resource requests for static pods and pods scheduled by kube-scheduler exceed the available resources of the node or if the total number of pods exceeds the maximum number of pods, one or more pods may fail with an OutOfCpu, OutOfMemory, or OutOfPods error.
8. daemonset-controller/replicaset-controller should notice that the number of healthy pods are not as expected and it will create new replicas. kube-scheduler will assign these new pods to the correct node and they should be running ultimately.
9. Pods with OutOfCpu, OutOfMemory, or OutOfPods error are cleaned up after sometime by pod garbage collector controller in kube-controller-manager based on how it is configured.

Why is this needed

This is a known issue kubernetes/kubernetes#115325 and generally happens on a node with static pods. kube-proxy is commonly run as a static pod too. Ultimately, this is not a harmful issue and only causes a delay in scheduling of pods correct.

We have ideas to fix this. But in the meantime, it would be a good idea to document this issue to educate the users and not spook them. We have had 38 bugs filed for such issues and there's little to no documentation about this failures.

Comments

[AnishShah]

/cc @tallclair

[sftim]

/sig node

[AnishShah]

/triage accepted

[sftim]

Help is wanted, although there's not enough detail in the issue description to justify adding /help to make it a bigger callout.

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted