Test Resource Limits for task submissions

[susheel]

• [ ] CPUs

• [x] RAM

• [ ] HDD

• [x] On submission

• [x] On Execution

[aniewielska]

In this task we are looking specifically at how cpu/memory/storage requests limits work on a pod level (so we don't take ResourcveQuotas into account - there is a separate issue for that). Docs: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/

Interesting bits about requests/limits:

1. Scheduling takes requests (both cpu/mem) into account (sum of requests cannot exceed node capacity)
2. Exceeding requests usually is permitted (but see below)
3. Exceeding CPU can cause throttling, exceeding mem limit will kill the container:

If a Container exceeds its memory request, it is likely that its Pod will be evicted whenever the node runs out of memory.

A Container might or might not be allowed to exceed its CPU limit for extended periods of time. However, it will not be killed for excessive CPU usage. 

4. If request of a new Pod would cause node capacity overflow, Pod is still created and stays in Pending.

Tests: Memory -> https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/

1. If pod mem request exceed node capacity - pod is not scheduled, but still created and gets Pending Status.
2. Pod's mem usage is allowed to exceed mem request (nothing happens, but it may change, if Node runs out of memory).
3. Container gets killed, when it's real usage exceeds memory limit.