clustermq
clustermq copied to clipboard
Support heterogeneous workers
Allow concatenation of different workers to a "pool."
Workflow logic is basically unchanged, because it does not matter where the workers are as long as they connect to the same socket.
This seems useful and might address my needs. I run into a situation where I have difficulty using the cluster efficiently. On my cluster the smallest unit is a 16 processes and I cant run long processes on the submitting machines. This means I until now was able to run everything within one node (master and workers using multicore). This however produces a scalability problem if i want to go to multiple nodes. In that case I could run a SLURM setup but this means that on the master node I only run one process while I would still have space for 15 workers. Combining local and slurm workers might solve this.