disBatch
disBatch copied to clipboard
flatironinstitute
Tool to distribute a list of computational tasks over a pool of compute resources. The pool can grow or shrink.
disBatch has worked great for me for dealing with heterogeneous tasks that take different amounts of CPU time. But I'm now facing a set of jobs with heterogeneous memory usage,...
Should we set common environment variables like `OMP_NUM_THREADS`, `MKL_NUM_THREADS`, etc. to the appropriate `-c` value? I've noticed a number of disbatch tasks each trying to use all the cores, so...
We should consider the following (at least one of which undoes a previous change): 1) Write to the driver.log file: - [ ] sys.argv - [ ] A dump of...
When a task fails, we currently continue. Allow different behaviors, such as exiting immediately, or exiting after running tasks finish, or just propagating exit code to disBatch somehow.
Should keep the status log, but can suppress other files if requested.
this would allow users to mix tasks with different threading requirements it would have to be an integer multiple of the current setting, and checked against the minimum workers/host
Metadata
Owner
Metadata
Tool to distribute a list of computational tasks over a pool of compute resources. The pool can grow or shrink.