fmriprep
fmriprep copied to clipboard
nipreps
BET fails due to a non-empty stderr although its return code is 0
What happened?
The fmriprep-reproducibility project evaluates the numerical stability of fmriprep. This project uses the fuzzy environment that adds random perturbations to floating-point operations. In addition, fuzzy prints debug information on the stderr during the execution. When executing BET with fuzzy, the debug information makes the stderr of BET non-empty, and fmriprep complains about it. However, the BET execution was successful (return code 0) and produced valid images.
What command did you use?
singularity run --pwd /workdir -B .:/workdir \
-B /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt \
-B /scratch/ychatel/fmriprep-reproducibility-experiments/outputs-anat-only/mca/27304881/openneuro/ds000256/derivatives/fmriprep/sub-CTS201/anat \
--env=VFC_BACKENDS="libinterflop_ieee.so" \
./envs/singularity-images/fmriprep-lts-fuzzy-20.2.1.sif \
--output-spaces MNI152NLin2009cAsym MNI152NLin6Asym \
--work-dir /workdir/fmriprep_work/27304881_28582650/ \
--anat-derivatives /scratch/ychatel/fmriprep-reproducibility-experiments/outputs-anat-only/m\
ca/27304881/openneuro/ds000256/derivatives/fmriprep/sub-CTS201/anat \
--notrack --write-graph --resource-monitor --omp-nthreads 1 --nprocs 1 --mem_mb 65536 \
--participant-label sub-CTS201 --random-seed 0 --skull-strip-fixed-seed \
--fs-license-file=/workdir/license.txt \
/workdir/inputs/openneuro/ds000256 \
/workdir/outputs-func-only/ieee/27304881_28582650/openneuro/ds000256/derivatives \
participant
What version of fMRIPrep are you running?
20.2.1
How are you running fMRIPrep?
Singularity
Is your data BIDS valid?
Yes
Are you reusing any previously computed results?
Anatomical derivatives
Please copy and paste any relevant log output.
211212-23:37:03,812 nipype.workflow INFO:
[Node] Running "skullstrip_first_pass" ("nipype.interfaces.fsl.preprocess.BET"), a CommandLine Interface with command:
bet /workdir/fmriprep_work/27304881_28582650/fmriprep_wf/single_subject_CTS201_wf/func_preproc_task_restbaseline_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct/ref_bold_corrected.nii.gz /workdir/fmriprep_work/27304881_28582650/fmriprep_wf/single_subject_CTS201_wf/\
func_preproc_task_restbaseline_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass/ref_bold_corrected_brain.nii.gz -f 0.20 -m
211212-23:37:05,38 nipype.workflow WARNING:
Storing result file without outputs
211212-23:37:05,41 nipype.workflow WARNING:
[Node] Error on "fmriprep_wf.single_subject_CTS201_wf.func_preproc_task_restbaseline_wf.initial_boldref_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_pass" (/workdir/fmriprep_work/27304881_28582650/fmriprep_wf/single_subject_CTS201_wf/func_preproc_task_restbaseline_w\
f/initial_boldref_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass)
211212-23:37:05,598 nipype.workflow ERROR:
Node skullstrip_first_pass failed to run on host blg9338.int.ets1.calculquebec.ca.
211212-23:37:05,599 nipype.workflow ERROR:
Saving crash info to /workdir/outputs-func-only/ieee/27304881_28582650/openneuro/ds000256/derivatives/fmriprep/sub-CTS201/log/20211212-223904_0f1aa0db-2d69-4c8b-83e4-2df6a2a61242/crash-20211212-233705-ychatel-skullstrip_first_pass-04e1d414-7ff4-4ddb-a7f0-298cb74a4459.txt
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
runtime = self._run_interface(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py", line 164, in _run_interface
self.raise_exception(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 745, in raise_exception
).format(**runtime.dictcopy())
RuntimeError: Command:
bet /workdir/fmriprep_work/27304881_28582650/fmriprep_wf/single_subject_CTS201_wf/func_preproc_task_restbaseline_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct/ref_bold_corrected.nii.gz /workdir/fmriprep_work/27304881_28582650/fmriprep_wf/single_subject_CTS201_wf/\
func_preproc_task_restbaseline_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass/ref_bold_corrected_brain.nii.gz -f 0.20 -m
Standard output:
Standard error:
Info [verificarlo]: loaded backend libinterflop_ieee.so
Info [verificarlo]: loaded backend libinterflop_ieee.so
Info [verificarlo]: loaded backend libinterflop_ieee.so
Info [verificarlo]: loaded backend libinterflop_ieee.so
Info [verificarlo]: loaded backend libinterflop_ieee.so
Info [verificarlo]: loaded backend libinterflop_ieee.so
Info [verificarlo]: loaded backend libinterflop_ieee.so
Info [verificarlo]: loaded backend libinterflop_ieee.so
Info [verificarlo]: loaded backend libinterflop_ieee.so
Info [verificarlo]: loaded backend libinterflop_ieee.so
Return code: 0
...
211211-05:08:17,405 nipype.workflow ERROR:
could not run node: fmriprep_wf.single_subject_CTS201_wf.func_preproc_task_restmovienofeedback_wf.final_boldref_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_pass
211211-05:08:17,405 nipype.workflow CRITICAL:
fMRIPrep failed: Workflow did not execute cleanly. Check log for details
211211-05:13:41,317 cli ERROR:
Preprocessing did not finish successfully. Errors occurred while processing data from participants: CTS201 (14). Check the HTML reports for details.
Additional information / screenshots
The singularity fuzzy image is available there https://github.com/glatard/fmriprep-lts-images/blob/db0826f0943d16135a966f28baa86c4da56e5b8d/fmriprep-lts-fuzzy-20.2.0.sif
I had another look, and apparently I completely missed this extremely obvious cause:
def _run_interface(self, runtime):
# The returncode is meaningless in BET. So check the output
# in stderr and if it's set, then update the returncode
# accordingly.
runtime = super(BET, self)._run_interface(runtime)
if runtime.stderr:
self.raise_exception(runtime)
return runtime
We could probably patch it to be something like:
def _run_interface(self, runtime):
# The returncode is meaningless in BET. So check the output
# in stderr and if it's set, then update the returncode
# accordingly.
runtime = super(BET, self)._run_interface(runtime)
if runtime.stderr:
if not all(line.startswith("Info ") for line in runtime.stderr.splitlines()):
self.raise_exception(runtime)
return runtime
Probably more general solutions are possible.
I can disable the debug info, so BET should not complain. But it would be nice to check the return code and the .stderr attribute to avoid this kind of issue.
Apparently at one point BET always returned 0 and was silent on success, so that's why stderr is used as the indicator. Looking at 5.0.7 source it looks like they do use EXIT_FAILURE, so it's not clear if there are still error conditions that don't result in error codes.
@pauldmccarthy Would you happen to know anything about this? Are we able to drop this check past a certain version of FSL?
Hi @effigies, indeed it seems that, before FSL 6.0.3, the bet script could return an exit code of 0 even if any of the underlying calls failed. This was fixed in 6.0.3.
On a related note, I've been gently trying to push to make our source repos at git.fmrib.ox.ac.uk public - hopefully this will happen in the near future..
