pygridtools
Add support for limiting the number of concurrently executing jobs
Currently there's no way to tell process_jobs (or consequently grid_map) that you only want a certain number of jobs to execute at a time. I think this should be fairly straightforward to add to the JobMonitor class by making it keep track of how many jobs are actively running and place jobs on hold or suspend them—I'm not sure what the difference in DRMAA is yet—when they request their inputs and function if the concurrent job limit is exceeded.
I've not read through all of the JobMonitor code, is there a reason that the jobs aren't submitted collectively as a job array? The latter versions of grid engine provide the -tc submission flag which limits the number of concurrently executing tasks of a job array. Submitting individual jobs seems quite integral to how everything works at the moment though.
@cjw85 The primary reason is that GridMap can be used with DRMs that don't support job arrays. We've tried to make as little of it Grid Engine specific as possible.