qbatch
qbatch copied to clipboard
pipitone
`--system local` is likely to be used semi-interactively on the command line. We should adjust the parallel call to be less chatty output wise, and maybe enable the progress bar...
The default of the submission filename means that the qbatch jobfiles get clobbered on every run, losing valuable history for debugging.
SLURM decided to be brain-dead and default to using minutes for unformatted numerical --time option, instead of seconds, like SGE/PBS/Torque. This means we need to be explicit about the supported...
I often want to wrap commands with ``/usr/bin/time -v`` to do benchmarking, it would convenient if I could just insert it at the front of every command
We should use the QBATCH prefix everywhere. Also it seems `nproc` will honour OMP_NUM_THREADS which means that many apps which rely on openmp will use proper number of threads even...
Problem arises like this QBATCH_CHUNKSIZE=40 QBATCH_PPJ=80 QBATCH_CORES=40 This results in THREADS_PER_COMMAND=2 regardless of the actual length of joblist. Instead, this should be adjusted for the length of the joblist if...
As per https://github.com/Mouse-Imaging-Centre/pydpiper/issues/350 - [ ] SGE - [ ] PBS - [ ] SLURM
qbatch needs an option where it only returns the jobids of jobs submitted. This would allow qbatch to be used in other pipelines that may want to monitor job IDs...
Parallel has a joblog function which will store exit codes as well as timings for all jobs. Should store that in /tmp and then cat it out at the end...