Spawn background processes across HPC cluster nodes

[angel-bee2018]

Good day! 😊

Is it currently possible for users to set up a callr command such as callr::r_bg to spawn background tasks across different nodes of a compute cluster? If not, could the existing workings behind callr::r_bg be by any chance easily applied across multiple nodes?

I've been trying to figure out how to make this potentially work on a large HPC cluster which runs on openmpi as the interface for inter-node communication. From the documentation, it seems like callr::r_bg is a wrapper for processx::process, which is itself based around calling system so it theoretically it should work right ...? however, I am not familiar enough with what callr::bg, for example, adds on top of the existing processx::process function that may make this harder or easier.

I would be interested to know any thoughts about this / suggested ways we can currently achieve the goal cheers!

[gaborcsardi]

processx::process does not use system. More importantly the IPC in processx is a Unix socket (or a Windows named pipe), so it is not going to work across machines.

[r2evans]

@angel-bee2018 I haven't done work on an OpenMPI cluster, but it seems that what you're asking for is more inline with the future package. There is a function makeClusterMPI in the parallely package (that is within the "futureverse"), perhaps that would help. (I don't know if or how to do what you need, but thought I'd provide a suggestion for your research.)