Workflow failed but not generating exit code?

[htwangtw]

What happened?

Not sure if this belongs to fMRIPrep or sMRIPrep.

I am processing a subject with invalid T1 file (details in Additional information). The job inevitably failed at the step zero of the anatomical workflow (t1w_ref_dimensions). It lead to an issue which nipype was outputting error log properly, but did not exit the program? I assume there was no exit code because my SLURM job stuck on the computing node (until I cancelled it).

What command did you use?

-w /DATA/fmriprep_work --participant-label 11104 --bids-filter-file /OUTPUT/bids_filters.json --task-id=rest --use-aroma  --output-spaces MNI152NLin2009cAsym MNI152NLin6Asym --output-layout bids --notrack --skip_bids_validation --write-graph --omp-nthreads 1 --nprocs 1 --resource-monitor /DATA/ds000030 /DATA/ds000030/derivatives/fmriprep participant

What version of fMRIPrep are you running?

20.2.1lts

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

Error setting node input:
Node: t1w_ref_dimensions
input: t1w_list
results_file: /DATA/fmriprep_work/fmriprep_wf/single_subject_11104_wf/bidssrc/result_bidssrc.pklz
value: /DATA/ds000030/sub-11104/anat/sub-11104_T1w.nii.gz

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/process.py", line 232, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 70, in run_node
    result["result"] = node.result
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 217, in result
    op.join(self.output_dir(), "result_%s.pklz" % self.name)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 291, in load_resultfile
    raise FileNotFoundError(results_file)
FileNotFoundError: /DATA/fmriprep_work/fmriprep_wf/single_subject_11104_wf/anat_preproc_wf/anat_template_wf/t1w_ref_dimensions/result_t1w_ref_dimensions.pklz
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
    callback(self)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
FileNotFoundError: /DATA/fmriprep_work/fmriprep_wf/single_subject_11104_wf/anat_preproc_wf/anat_template_wf/t1w_ref_dimensions/result_t1w_ref_dimensions.pklz
slurmstepd: error: *** JOB 5377944 ON nc10336 CANCELLED AT 2022-05-05T20:24:43 ***

Additional information / screenshots

I am processing ds000030. The T1w image of one subject (sub-11104) was not correctly fetched by datalad. I processed the subject because I didn't read the error message. BIDS validator did not catch it.

The slurm script was generate through fmriprep-slurm