Support container creation when resource quotas are hit

[Dutchy-]

Currently, when kubedock creates pods and you hit a resource quota, the request fails with this error:

{"message":"pods \"kubedock-a4a0dbae1bd2\" is forbidden: exceeded quota: tenant-quota, requested: limits.cpu=2, used: limits.cpu=31650m, limited: limits.cpu=32"}

Context: we run kubedock as a sidecontainer on Tekton to support java testcontainers.

Is it possible to support creating pods in this situation without having the request fail immediately?

Some solutions I considered:

• Use a Deployment instead of a Pod to schedule the container
• Use a retry mechanism

And maybe there are other options?

[joyrex2001]

In earlier versions kubedock was using deployments, and had the option to use jobs. Deployments were problematic when the orchestrated containers were actually one-off jobs (jobs did solve that).

A retry option might make sense and probably leads to increased succes when orchestrating, but tests might still fail because of time-outs in the actual tests as well.

Something that might work already, is setting lower requests and limits (via labels, global settings, or a global pod template).

[Dutchy-]

Yes, lowering requests and limits relieves the problem temporarily, and it's definitely something we're doing now to work around the problem, but it does not solve the problem permanently - even on lower limits the pod might hit the quota and get denied.