Protect kubelet against PID exhaustion

[llarsson]

Is your feature request related to a problem? Please describe.

The NSA/CISA Kubernetes Hardening Guide has the following to say about PID exhaustion:

Process IDs (PIDs) are a fundamental resource on nodes and can be exhausted without violating other resource limits. PID exhaustion prevents host daemons (such as kubelet and kube-proxy) from running. Administrators can use node PID limits to reserve a specified number of PIDs for system use and Kubernetes system daemons. Pod PID limits are used to limit the number of processes running on each Pod. Eviction policies can be used to terminate a Pod that is misbehaving and consuming abnormal resources. However, eviction policies are calculated and enforced periodically and do not enforce the limit.

The kubelet can be protected against this with settings.

Describe the solution you'd like

I'd like the solution that we make PID limiting both configurable and that we put a reasonable limit in place, to avoid exhaustion. Ideally, I would want there to be a setting for Nodes and one for Pods, since that's what kubelet is set up to support.

Describe alternatives you've considered

Not doing it. Which is what we are currently doing, since upstream kubespray doesn't expose this setting. See also https://github.com/kubernetes-sigs/kubespray/issues/5535

But that seems like a rather needless risk.

Additional context

N/A

Definition of done:

• There are settings in the KubeletConfiguration configuration that we can use for this purpose.
• There are default values that Makes Sense (TM), set to something conservative for both Nodes and Pods. Note that some investigation could be required here to determine good values.

Stretch goal: upstream this to Kubespray, and close https://github.com/kubernetes-sigs/kubespray/issues/5535.