Ability to prioritise tasks on the Kubernetes Glue

[hanifa2102]

Currently, all tasks pushed into the kubernetes glue are executed in uniform manner, meaning there is no way to prioritise different tasks at different rates.

python k8s_glue_example.py --queue singleQueue

Contrasting this with running clearml-agent, High and Low priority Queues

clearml-agent daemon --queue highQueue lowQueue

Round Robin Priority Queues

cleamrl-agent daemon --queue firstQueue secondQueue --order-fairness

[bmartinn]

Hi @hanifa2102 I think if we introduce instance limit, this will infer the priority of the jobs. Basically as jobs are pulled by order, they are pushed into the k8s, then if we hit the max k8s instance limit, we stop pulling jobs until a k8s job is completed, then we continue. wdyt ?