Autorecon1 Crashing

[smeisler]

What happened?

Autorecon 1 appears to be failing and not providing a helpful error message.

What command did you use?

singularity run -e -B ${scratch} $IMG ${scratch}/${subject}_db ${scratch}/${subject}_db/derivatives  \ 
participant --participant_label ${subject:4} -w $scratch/${subject}_work \ 
--fs-license-file  ${scratch}/${subject}_db/code/license.txt --output-layout legacy \
--skip-bids-validation --stop-on-first-crash -t $task  \
--fd-spike-threshold .9 --dvars-spike-threshold 3 --mem 31500  \
--output-spaces T1w MNI152NLin6Asym:res-2

What version of fMRIPrep are you running?

22.0.0rc0

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

[Node] Executing "autorecon1" <smriprep.interfaces.freesurfer.ReconAll>
220528-11:48:40,992 nipype.workflow INFO:
	 [Node] Finished "autorecon1", elapsed time 0.77419s.
220528-11:48:41,7 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_22_0_wf.single_subject_NDARAA306NT2_wf.anat_preproc_wf.surface_recon_wf.autorecon1" (/om2/scratch/Sat/smeisler/HBN_DM_SCRATCH/sub-NDARAA306NT2_work/fmriprep_22_0_wf/single_subject_NDARAA306NT2_wf/anat_preproc_wf/surface_recon_wf/autorecon1)
220528-11:48:42,834 nipype.workflow ERROR:
	 Node autorecon1 failed to run on host node039.
220528-11:48:42,845 nipype.workflow ERROR:
	 Saving crash info to /om2/scratch/Sat/smeisler/HBN_DM_SCRATCH/sub-NDARAA306NT2_db/derivatives/fmriprep/sub-NDARAA306NT2/log/20220528-114715_003f85be-839f-44cd-8d1f-fe81ffec540d/crash-20220528-114842-smeisler-autorecon1-e0ad7322-dd4c-4a03-9b22-70f6614a2539.txt
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon1.

RuntimeError: subprocess exited with code 1.

220528-11:48:42,895 nipype.workflow CRITICAL:
	 fMRIPrep failed: Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon1.

RuntimeError: subprocess exited with code 1.

220528-11:48:43,994 cli ERROR:
	 Preprocessing did not finish successfully. Errors occurred while processing data from participants: NDARAA306NT2 (1). Check the HTML reports for details.
220528-11:48:43,994 cli ERROR:
	 Preprocessing did not finish successfully. Errors occurred while processing data from participants: NDARAA306NT2 (1). Check the HTML reports for details.

Additional information / screenshots

Each subject is its own job, 32GB 8CPU

Not all subjects have this error, but all subjects came from the same dataset. Looking now I cannot think of anything that discerns the crashing vs passing subjects.

Centos7.6

From the crash file mentioned in the log output:

Node: fmriprep_22_0_wf.single_subject_NDARAA306NT2_wf.anat_preproc_wf.surface_recon_wf.autorecon1
Working directory: /om2/scratch/Sat/smeisler/HBN_DM_SCRATCH/sub-NDARAA306NT2_work/fmriprep_22_0_wf/single_subject_NDARAA306NT2_wf/anat_preproc_wf/surface_recon_wf/autorecon1

Node inputs:

FLAIR_file = <undefined>
T1_files = <undefined>
T2_file = <undefined>
args = <undefined>
big_ventricles = <undefined>
brainstem = <undefined>
directive = autorecon1
environ = {}
expert = <undefined>
flags = <undefined>
hemi = <undefined>
hippocampal_subfields_T1 = <undefined>
hippocampal_subfields_T2 = <undefined>
hires = <undefined>
mprage = <undefined>
mri_aparc2aseg = <undefined>
mri_ca_label = <undefined>
mri_ca_normalize = <undefined>
mri_ca_register = <undefined>
mri_edit_wm_with_aseg = <undefined>
mri_em_register = <undefined>
mri_fill = <undefined>
mri_mask = <undefined>
mri_normalize = <undefined>
mri_pretess = <undefined>
mri_remove_neck = <undefined>
mri_segment = <undefined>
mri_segstats = <undefined>
mri_tessellate = <undefined>
mri_watershed = <undefined>
mris_anatomical_stats = <undefined>
mris_ca_label = <undefined>
mris_fix_topology = <undefined>
mris_inflate = <undefined>
mris_make_surfaces = <undefined>
mris_register = <undefined>
mris_smooth = <undefined>
mris_sphere = <undefined>
mris_surf2vol = <undefined>
mrisp_paint = <undefined>
openmp = 8
parallel = <undefined>
steps = <undefined>
subject_id = recon_all
subjects_dir = <undefined>
talairach = <undefined>
use_FLAIR = <undefined>
use_T2 = <undefined>
xopts = <undefined>

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon1.

RuntimeError: subprocess exited with code 1.

[effigies]

This is weird. The fact that subjects_dir is <undefined> makes no sense.

Just to confirm, you're using a fresh scratch directory and not pre-run FreeSurfer?

[smeisler]

Yes, no previous outputs and fresh working directory. What's also weird is that sMRIPrep ran fine on these subjects before (0.8.1 , so FS v6). To be clear these outputs are not available for the run I am doing currently, and I am having fMRIPrep do FS 7.2 from scratch.

[effigies]

Are you able to replicate this by running sMRIPrep 0.9.0?

[marlietandoc]

I am having the same issue using Docker and cannot get past it (I only have 1 subjects worth of data at the moment).

My command

docker run --mount type=bind,source=${workdir},target=/workdir nipreps/fmriprep \
  --participant-label sub-001 \
  --fs-license-file /workdir/code/license.txt \
  --work-dir /workdir/work \
  --stop-on-first-crash \
  --fs-no-reconall \
  -vvv \
  --omp-nthreads 16 \
  /workdir/bids /workdir/bids/derivatives/fmriprep participant

Crash file from the log output

Node: fmriprep_22_0_wf.single_subject_001_wf.anat_preproc_wf.surface_recon_wf.autorecon1
Working directory: /workdir/work/fmriprep_22_0_wf/single_subject_001_wf/anat_preproc_wf/surface_recon_wf/autorecon1

Node inputs:

FLAIR_file = <undefined>
T1_files = <undefined>
T2_file = <undefined>
args = <undefined>
big_ventricles = <undefined>
brainstem = <undefined>
directive = autorecon1
environ = {}
expert = <undefined>
flags = <undefined>
hemi = <undefined>
hippocampal_subfields_T1 = <undefined>
hippocampal_subfields_T2 = <undefined>
hires = <undefined>
mprage = <undefined>
mri_aparc2aseg = <undefined>
mri_ca_label = <undefined>
mri_ca_normalize = <undefined>
mri_ca_register = <undefined>
mri_edit_wm_with_aseg = <undefined>
mri_em_register = <undefined>
mri_fill = <undefined>
mri_mask = <undefined>
mri_normalize = <undefined>
mri_pretess = <undefined>
mri_remove_neck = <undefined>
mri_segment = <undefined>
mri_segstats = <undefined>
mri_tessellate = <undefined>
mri_watershed = <undefined>
mris_anatomical_stats = <undefined>
mris_ca_label = <undefined>
mris_fix_topology = <undefined>
mris_inflate = <undefined>
mris_make_surfaces = <undefined>
mris_register = <undefined>
mris_smooth = <undefined>
mris_sphere = <undefined>
mris_surf2vol = <undefined>
mrisp_paint = <undefined>
openmp = 8
parallel = <undefined>
steps = <undefined>
subject_id = recon_all
subjects_dir = <undefined>
talairach = <undefined>
use_FLAIR = <undefined>
use_T2 = <undefined>
xopts = <undefined>

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon1.

RuntimeError: subprocess exited with code 1.

[mafortin]

Exact same issue for me. Looking for a solution! :/

[mgxd]

is there any information in the log file produced by freesurfer? (found in <subjects-dir>/<subject>/scripts/recon-all.log)

[mafortin]

Weirdly it seems to be a log file from another calculation/person/run... I'm not sure to understand so here is mine: """ Tue Apr 28 01:24:00 EDT 2009 /autofs/space/minerva_001/users/nicks/dev/distribution/average/fsaverage /space/minerva/1/users/nicks/freesurfer/bin/recon-all -s fsaverage -cortparc subjid fsaverage setenv SUBJECTS_DIR /autofs/space/minerva_001/users/nicks/dev/distribution/average FREESURFER_HOME /space/minerva/1/users/nicks/freesurfer Actual FREESURFER_HOME /autofs/space/minerva_001/users/nicks/freesurfer build-stamp.txt: freesurfer-x86_64-redhat-linux-gnu-dev4-20090216 Linux minerva 2.6.9-78.0.13.ELsmp #1 SMP Wed Jan 14 15:55:36 EST 2009 x86_64 x86_64 x86_64 GNU/Linux cputime unlimited filesize unlimited datasize unlimited stacksize 10240 kbytes coredumpsize unlimited memoryuse unlimited vmemoryuse unlimited descriptors 1024 memorylocked 32 kbytes maxproc 65536

         total       used       free     shared    buffers     cached

Mem: 7403128 6807536 595592 0 133040 6185568 Swap: 16386292 224 16386068

######################################## program versions used [...] """

It continues for a lot longer so I attached the complete recon-all.log file.

recon-all.log

[smeisler]

At least on my end (using fMRIPrep), a <FSsubjects-dir>/<subject> directory was not even created. The fsaverage directory is there.

[mafortin]

Yeah me too in fact, this file is inside fsaverage/scripts folder.

And just to be clear, I said it looks like a run from "someone else" because the linux setup, SUBJECTS_DIR, FREESURFER_HOME, memory available, etc. are not mine... which confuses me a lot.

[effigies]

Could this be a --cleanenv issue, with external FREESURFER_HOME overriding our settings?

[mafortin]

--cleanenv is not a recognized argument for smriprep-docker

[smeisler]

--cleanenv is not a recognized argument for smriprep-docker

--cleanenv is a singularity option (not a smriprep/fmriprep argument) which makes sure none of your local environmental variables are brought into the container.

[mafortin]

--cleanenv is not a recognized argument for smriprep-docker

--cleanenv is a singularity option (not a smriprep/fmriprep argument) which makes sure none of your local environmental variables are brought into the container.

Yes, sorry for the mistake. I am using Docker so I don't have this option. :)

[marlietandoc]

At least on my end (using fMRIPrep), a <FSsubjects-dir>/<subject> directory was not even created. The fsaverage directory is there.

Same on my end too that the directory was not created.

[mgxd]

Ok, looking at the log a little closer, autorecon1 instantly fails:

[Node] Executing "autorecon1" <smriprep.interfaces.freesurfer.ReconAll>
220528-11:48:40,992 nipype.workflow INFO:
	 [Node] Finished "autorecon1", elapsed time 0.77419s.

Perhaps the best way to debug this would be to:

1. Shell into the container (either with the --shell option on the docker wrapper, or singularity shell), being sure to include all the original mounts
2. Running the command located in the file (hopefully it exists)

<workdir>/fmriprep_22_0_wf/single_subject_<subject>_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt

[effigies]

Hi all, just checking in, has anybody been able to follow @mgxd's instructions and get some more information here?

[smeisler]

I have not (my scratch directory was in temporary storage that got wiped) but I will work on replicating this.

[mafortin]

I haven't tested it yet. How would I implement this for smriprep-docker?

[mgxd]

If you keep all the options you previously used, and add in --shell, you should then be dropped in to an interactive session on the terminal. If the working directory is the same, you can run the following command, making sure to replace <subjectid> with your subject.

cat /scratch/smriprep_wf/single_subject_<subjectid>_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt | bash

[mafortin]

Ok thank you @mgxd ! :) I'm currently on a work trip, so I'll test it out ASAP next week when I'll comeback.

[marlietandoc]

Thanks for looking into this everyone. I found that I no longer get this error when I use the fmriprep-docker wrapper instead of the command I was using above using docker directly. I am not sure why this worked. I wasn't sure how to shell in when I wasn't using the wrapper (--shell wasn't a recognized argument).

[effigies]

I wasn't sure how to shell in when I wasn't using the wrapper (--shell wasn't a recognized argument).

You can add --entrypoint=bash to the docker arguments and remove all fMRIPrep arguments.

[smeisler]

Hello,

I ran the command in <workdir>/fmriprep_22_0_wf/single_subject_<subject>_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt:

INFO: hi-res volumes are conformed to the min voxel size
fsr-checkxopts: Command not found.
Linux openmind7 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-inside7003 exited with ERRORS at Fri Jun 10 19:53:28 EDT 2022

For more details, see the log file 
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Tab autocomplete in the singularity shell revealed the only command in PATH that begins "fsr" is fsr-getxopts.

[smeisler]

I mounted my local version of fsr-checkxopts (from FS6) to the container and it appears to be progressing.

[effigies]

Good find. That should be easy to fix in the next release.

[mafortin]

Hi! sorry for being late, but the issue should be fixed now based on what I read here? However, being a newbie with sMRIPrep, what should I do to use the "debugged" version? Re-pull the Docker Image? Thanks in advance! :)

[effigies]

@mafortin I've just tagged sMRIPrep 0.9.1 and fMRIPrep 22.0.0rc1. When those images hit DockerHub, you should be good to go.

[mafortin]

FYI: smriprep-docker 0.9.1 ran for one of the two subjects but crashed with the same error code as before for the 2nd subject.

[smeisler]

This may be because, in a similar fashion, getfullpath is not in the docker container, but may be called by FreeSurfer.

[mgxd]

Thank you @smeisler for tracking that down!