clustermq
clustermq copied to clipboard
Add options for max memory, n_jobs
One of the ideas of the option-based configuration of clustermq
is that code should be portable between compute environments, e.g. when moving to a different HPC scheduler.
If moving from HPC to local parallelization (even on a heavy duty server), resources will be much more limiting.
For a multicore setup, it would be nice to provide options for setting limits on total concurrent jobs (defaulting to cores/available cores) and memory (defaulting to memory available on that machine)
This way HPC scripts would not need to be adjusted, but could run (at reduced capacity) on multicore environments