CuBIDS icon indicating copy to clipboard operation
CuBIDS copied to clipboard

parallelize sequential flag with drmaa or multiple cores

Open Terf opened this issue 3 years ago • 1 comments

Submitting validator jobs to the cluster (in batches) provides massive speedup, running on multiple cores provides more modest speedup. one drawback is that the progress bar is kinda meaningless since jobs are run asynchronously, and when qsubing jobs a bunch of tempfiles need to be created in a networked-mounted directory (uses bids_dir) so they're visible to the exec nodes, which is a bit hacky.

Terf avatar Aug 09 '21 18:08 Terf

also when running on CUBIC you'll see the output

User-specified core binding strategy: linear
User-specified core binding type: set
User-specified core binding amount: 1

for each job submitted which kinda nukes your terminal but Mark said it's just bc he was debugging the scheduler and will take them out

Terf avatar Aug 09 '21 18:08 Terf