adding compatibility to other cluster job scheduler such as TORQUE

[hengwangxian]

Hi, Other than SGE, is it possible to add to pyflow compatibility to clusters running other job schedulers such as PBS (TORQUE)? Thanks!

[ctsa]

Thanks Heng,

In principal this is mostly a matter of splitting the current "SGETaskRunner" class into a parent with two children, and setting up the equivalent commandline calls for another scheduler.

This ticket should serve as a placeholder for any such effort (none is active).

As we discussed the 'larger' solution to this problem would probably be to add a DRMAA option. I'll add an issue to track notes on that possibility, although there is also no active effort to address it.

[jjfarrell]

Hi @ctsa

Lots of academic institutions using SGE have or are planning to migrate to SLURM. This is lately driven since SGE is now no longer supported. Is there any news for adding other schedulers? We are using pyflow with Manta software. If this effort is not active, would it feasible to migrate the Manta workflow to something like NextFlow. Is there a description of the Manta workflow somewhere?