ThreadPools.jl
ThreadPools.jl copied to clipboard
[Question] Does `qmap` work for multithreads of cross workers (on cluster)?
Nice work! qmap
efficiently solved the nonuniform-load task problems. Now I want to ask a deeper question. I have 2 nodes (or any HPC platforms) connecting to each other by ssh. I have verified this by
addprocs([("user@node02", 1)], exeflags = ["-t 16"])
addprocs([("user@node03", 1)], exeflags = ["-t 32"])
After executing the codes above, I obtain 2 workers [2, 3]
. worker 2 is opened on node02 and has 16 threads while worker 3 is opened on node03 and has 32 threads. So totally I have 48 threads to execute nonuniform-load tasks. I found this ThreadPools.jl seems to only support qmap
in a single node, which means only one node is busy calculating and other nodes are idling. Is there anything I missed, or is there any Pkg I did not notice?
Any discussions are welcome since the author of ThreadPools.jl has been inactive for more than two years...