Vectorize Nipype

[JBarsotti]

Hello,

I hope you are well!

Is it possible to vectorize nipype instead of using SGEGraph with it? I would like to avoid using a Sun Grid Engine if I can help it.

Thanks!

John

[JBarsotti]

Sorry for the confusion. I meant to ask if I could convert SGEGraph to an array-type job submission. The reason is because the HPC system I am running it on, Argon, is having trouble with SGEGraph. I have attached the way the problem has been explained below:

"What is happening is that SGE has more to do than it can handle in the allotted time that is has for processing jobs per scheduling cycle. This has been triggered by an uptick in the number of jobs being submitted via scripts, presumably in a loop. While it is natural to write a script to submit a 100 or a 1000 jobs, when several such scripts are running at the same time, the rate of job submission overwhelms SGE. SGE will spend most of its cycles trying to submit the jobs, but eventually it has to break from that to schedule jobs to run on the system. Obviously, a high rate of job submissions also produces a large number of jobs that have to be processed. This makes the scheduler thread take a longer time to complete. Meanwhile, more jobs are coming in at a high rate, and a snow ball effect begins, eventually leading to time outs, and failed commands, including qsub. SGE will prioritize scheduling over other events, so jobs are still being scheduled even though commands are timing out.

The only solution to this is to reduce the rate of job submission. The best way to do that is to use array jobs as those can reduce, for example, 1000 job submissions down to a single job submission. If the jobs do not have complex dependencies then it is usually possible to create a task file that will contain the list of computations, and submit the task file as an array job."

[satra]

@JBarsotti - presently the submission of job arrays is not possible in nipype. however, if a running job is allowed to submit new jobs on your system, you can use the SGE plugin and set the max_jobs parameter (https://miykael.github.io/nipype_tutorial/notebooks/basic_plugins.html#Using-Nipype-Plugins), and Nipype will control the number of jobs it runs at any given moment using it's own scheduler. also job arrays as noted above are typically used in instances of no dependencies. a nipype graph, except for the simplest use case of a single node graph, typically has dependent nodes.

[JBarsotti]

Thank you so much for your reply, Satra!

I will give that a shot and see how it goes.

John