Support container creation when resource quotas are hit

[Dutchy-]

Currently, when kubedock creates pods and you hit a resource quota, the request fails with this error:

{"message":"pods \"kubedock-a4a0dbae1bd2\" is forbidden: exceeded quota: tenant-quota, requested: limits.cpu=2, used: limits.cpu=31650m, limited: limits.cpu=32"}

Context: we run kubedock as a sidecontainer on Tekton to support java testcontainers.

Is it possible to support creating pods in this situation without having the request fail immediately?

Some solutions I considered:

• Use a Deployment instead of a Pod to schedule the container
• Use a retry mechanism

And maybe there are other options?