clustermq icon indicating copy to clipboard operation
clustermq copied to clipboard

Add options for max memory, n_jobs

Open mschubert opened this issue 3 years ago • 0 comments

One of the ideas of the option-based configuration of clustermq is that code should be portable between compute environments, e.g. when moving to a different HPC scheduler.

If moving from HPC to local parallelization (even on a heavy duty server), resources will be much more limiting.

For a multicore setup, it would be nice to provide options for setting limits on total concurrent jobs (defaulting to cores/available cores) and memory (defaulting to memory available on that machine)

This way HPC scripts would not need to be adjusted, but could run (at reduced capacity) on multicore environments

mschubert avatar Nov 19 '20 19:11 mschubert