clustermq icon indicating copy to clipboard operation
clustermq copied to clipboard

Support heterogeneous workers

Open mschubert opened this issue 5 years ago • 1 comments

Allow concatenation of different workers to a "pool."

Workflow logic is basically unchanged, because it does not matter where the workers are as long as they connect to the same socket.

mschubert avatar May 10 '19 16:05 mschubert

This seems useful and might address my needs. I run into a situation where I have difficulty using the cluster efficiently. On my cluster the smallest unit is a 16 processes and I cant run long processes on the submitting machines. This means I until now was able to run everything within one node (master and workers using multicore). This however produces a scalability problem if i want to go to multiple nodes. In that case I could run a SLURM setup but this means that on the master node I only run one process while I would still have space for 15 workers. Combining local and slurm workers might solve this.

bart1 avatar Sep 14 '20 10:09 bart1