Autoalloc improvements

[spirali]

Improve the existing implementation of autoallocation.

• [x] #128

• [x] #129

• [x] #145

• [ ] Match HQ workers with allocations. Currently, we have no information about which HQ workers belong to which allocation. We should match allocations with workers automatically, display this information to the user and also use it in the autoallocator itself (for example, if an allocation runs, but no worker connects, there is some problem).

  It would be nice to implement some generic "tagging" system for workers, for example by attaching a set of key=value tags to a worker and then use it for this matching. Later it could also be used for resource management (e.g. run this job on a worker with a GPU).

• [ ] Improve the auto-allocation logic:

• Currently it only auto-scales a certain number of workers. It should also take tasks into account, and spawn workers based on the task amount.

• If several errors have happened in a row, stop trying to allocate more jobs and print some warning to the user.

• [ ] #174

[giovannipizzi]

Is it also possible to specify a maximum number of workers allowed (or, similarly, a maximum number of jobs allowed in the queue, even in total for all allocs?) Most supercomputers have a hard limit on the number of jobs in the scheduler, so I'd like to be able to submit thousands of jobs to hq and `hq will process them without submitting more than X workers to the queue, where e.g. X could be of the order of 100 and should be configurable by the user

[Kobzol]

Hi! We were thinking of including such a limit (it wouldn't be hard to implement), but then we thought that it might not be needed, since as you said, the HPC queue system will stop HQ sooner or later. We consider that to be the optimal scenario, since HQ would then use as many resources as it can be provided.

What would be the use case for this additional worker limit? Limiting the number of workers spawned by HQ before it gets limited by PBS/Slurm, so that you can e.g. you the remaining part of the limit for unrelated allocations?

[giovannipizzi]

Indeed, I had 3 scenarios:

• just not have hq alloc keep trying submitting and failing, but just stopping submitting after a certain amount; however if hq alloc deals nicely with errors due to the queue size and doesn't start resubmitting very quickly, this is ok
• have multiple allocs, and limit the number of workers in each to decide how to balance the load (again here I might be misunderstanding how multiple allocs are supposed to be used)
• indeed, keep part of the cluster for other non-hq users - e.g. one could submit multi-node jobs directly to SLURM, and smaller sub-node jobs to hq, and e.g. ask hq to avoid running more than half the allowed jobs

[Kobzol]

I see. Ok, that makes sense, we can add a hard limit of workers spawned by an allocation queue. The use case for multiple allocation queues is currently meant for scenarios where you want to use e.g. two projects or two queues together. For example:

# Use qexp as much as possible
$ hq add pbs --timelimit 1h -- -q qexp
# But also add resources from some accounted project
$ hq add pbs --timelimit 4h -- -q qprod -A ProjectX

You can also use multiple allocation queues to spawn different types of workers (e.g. workers with GPU).

[Kobzol]

The worker limit was implemented in https://github.com/It4innovations/hyperqueue/pull/262.