Conventional schedulers (Slurm/PBS/Loadleveler) compatibility?

[surak]

Is your feature request related to a problem? Please describe. Most supercomputers in the world use of the few schedulers available, like the ones mentioned at the title. Those usually don't play well with other schedulers.

Describe the solution you'd like To be able to run it directly from a slurm session.

Describe alternatives you've considered In many SPMD environments, one is able to submit one single program to run in a number of compute nodes. So, this program should be able to run as a master in one node, and as a worker in all the other nodes. Something like a front-end to mars.

Additional context Supercomputer schedulers are quite simple in operation. The user submits a job to the batch system, which waits until the amount of resources requested is available. Then, it runs the code in all the processes. It's that simple.

[qinxuye]

As current developers of Mars don't have such background or experience on supercomputers. We may need the help from the community. Actually It will be fantastic if you may try to deploy Mars and it's runtime to such an environment, with contributing back the code, we'll appreciate that so much.

[raybellwaves]

Worth taking a look at https://github.com/dask/dask-jobqueue

[chaokunyang]

Can we add other scheduler like Slurm here using execution API? @fyrestone

[fyrestone]

Can we add other scheduler like Slurm here using execution API? @fyrestone

It seems that this is a deployment problem, not an execution issue.