ThreadPools.jl icon indicating copy to clipboard operation
ThreadPools.jl copied to clipboard

[Question] Does `qmap` work for multithreads of cross workers (on cluster)?

Open draftman9 opened this issue 4 months ago • 0 comments

Nice work! qmap efficiently solved the nonuniform-load task problems. Now I want to ask a deeper question. I have 2 nodes (or any HPC platforms) connecting to each other by ssh. I have verified this by 

addprocs([("user@node02", 1)], exeflags = ["-t 16"])
addprocs([("user@node03", 1)], exeflags = ["-t 32"])

After executing the codes above, I obtain 2 workers [2, 3]. worker 2 is opened on node02 and has 16 threads while worker 3 is opened on node03 and has 32 threads. So totally I have 48 threads to execute nonuniform-load tasks. I found this ThreadPools.jl seems to only support qmap in a single node, which means only one node is busy calculating and other nodes are idling. Is there anything I missed, or is there any Pkg I did not notice?

Any discussions are welcome since the author of ThreadPools.jl has been inactive for more than two years...

draftman9 avatar Oct 13 '24 08:10 draftman9