Task optimization in Linux

[stsour]

Hi!

I am working in an HPC environment and want to know if it is possible to split the various tasks into separate jobs. For example, I have already run tasks 1-3 on a dataset, but ran into a compute time limit. Is there a way I can direct the software to start with task 4 and use results from a previous job? I.e. the timestamped directory that is created that contains the task settings and results.

I am working with rather large datasets and I understand the software is not yet optimized for parallelization. Do you have any recommendations as to optimizing the processing of the various tasks?

Thanks, Shiri

[rmillikin]

Apologies for the late reply. There's no automatic option to start from a previous task, but you can use the -o flag to specify an output directory. For example you could run a calibration task with a particular output folder, and that single task is the only task for that run. Then start another run, pointing it to the calibrated data files from the previous task. Does that make sense?