CuBIDS
CuBIDS copied to clipboard
parallelize sequential flag with drmaa or multiple cores
Submitting validator jobs to the cluster (in batches) provides massive speedup, running on multiple cores provides more modest speedup. one drawback is that the progress bar is kinda meaningless since jobs are run asynchronously, and when qsubing jobs a bunch of tempfiles need to be created in a networked-mounted directory (uses bids_dir) so they're visible to the exec nodes, which is a bit hacky.
also when running on CUBIC you'll see the output
User-specified core binding strategy: linear
User-specified core binding type: set
User-specified core binding amount: 1
for each job submitted which kinda nukes your terminal but Mark said it's just bc he was debugging the scheduler and will take them out