abcd-hcp-pipeline icon indicating copy to clipboard operation
abcd-hcp-pipeline copied to clipboard

Published 20 hours ago • verified publisher icon DCAN-Labs

Stuck at FreeSurfer Stage (mris_fix_topolo)

Open fullcream9 opened this issue 7 months ago • 2 comments

What happened?

Dear abcd-hcp-pipeline Developers,

I am currently working on processing Human Brain Network (HBN) subjects and have encountered an issue with the abcd-hcp-pipeline version 0.1.3, specifically, I am stuck at the FreeSurfer stage. Specifically, the process seems to hang indefinitely during the execution of the "mris_fix_topolo" command. This issue is observed through the top command, which shows continuous execution without progression.

I am tasked with processing over 400 subjects and have successfully processed 286 of them before encountering this problem.

What command did you use?

cat HBN_BIDS_list_1108.txt | parallel --jobs 26 docker run --rm -v /home/panshi/ABCD_Results/panshi/HBN/ABCD_HCP_pipe/ABCD-BIDS/CDODD_BIDS_raw:/bids_input:ro -v /home/panshi/Documents/ABCD-BIDS/HBN_deri_temp/:/output -v /home/panshi/Downloads/freesurfer_license/:/license dcanumn/abcd-hcp-pipeline /bids_input /output --freesurfer-license=/license/license.txt --ncpus 3 --participant-label {} >> parallel_docker_HBN_1108_newserver.log 2>&1

What version of the abcd-hcp pipeline are you using?

abcd-hcp-pipeline v0.1.3

Directory Structure

No response

Relevant log output

abcd-hcp-pipeline v0.1.3
running FreeSurfer
/opt/pipeline/FreeSurfer/FreeSurferPipeline.sh \
    --subject=NDARRR570MC0 \
    --subjectDIR=/output/sub-NDARRR570MC0/ses-None/files/T1w \
    --t1=/output/sub-NDARRR570MC0/ses-None/files/T1w/T1w_acpc_dc_restore.nii.gz \
    --t1brain=/output/sub-NDARRR570MC0/ses-None/files/T1w/T1w_acpc_dc_restore_brain.nii.gz \
    --t2=/output/sub-NDARRR570MC0/ses-None/files/T1w/T2w_acpc_dc_restore.nii.gz \
    --useT2=true \
    --printcom=
missing expected outputs from FreeSurfer
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/T1w_hires.nii.gz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/lh.EC_average
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/rh.EC_average
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/lh.aparc.a2009s.annot
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/lh.aparc.annot
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/rh.aparc.annot
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/T1w_hires.masked.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aparc+aseg.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aparc.a2009s+aseg.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aseg.hires.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aseg.hires.nii.gz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/brain.hires.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/lh.ribbon.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/rh.ribbon.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/ribbon.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/wm.hires.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/wmparc.mgz
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/aseg.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.aparc.a2009s.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.aparc.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.curv.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.aparc.a2009s.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.aparc.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.curv.stats
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.mid
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.prehires
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv.prehires
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sphere
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sphere.reg
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sulc
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.thickness
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.volume
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white.deformed
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white.prehires
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area.mid
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.curv
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.curv.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.inflated
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.orig
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.pial
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sphere
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sphere.reg
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sulc
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.thickness
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.volume
file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.white
Traceback (most recent call last):
  File "/app/run.py", line 397, in <module>
    _cli()
  File "/app/run.py", line 69, in _cli
    return interface(**kwargs)
  File "/app/run.py", line 393, in interface
    stage.run(ncpus)
  File "/app/pipelines.py", line 592, in run
    self.teardown(result)
  File "/app/pipelines.py", line 538, in teardown
    self.__class__.__name__)
Exception: error caught during stage: FreeSurfer

        The pipeline must choose distortion correction method based on the
        type(s) of field maps available. The type of fieldmaps you have are
        either not able to be used in the abcd-hcp-pipeline or they are
        not properly identified in the BIDS format. The pipeline does not
        account for 'phasediff', 'magnitude', and 'fieldmap' field maps
        filetypes yet. If you have 'phasediff' and 'magnitude' field maps,
        please provide the original 'phase1', 'phase2', 'magnitude1', and
        'magnitude2' field maps used to calculate those files. The pipeline
        does the calculation itself.

/home/panshi/Documents/ABCD-BIDS/HBN_deri_temp/sub-NDARBE096YK6/ses-None/logs/FreeSurfer/status.json
{
    "num_runs": 1,
    "node_status": 3,
    "comment": "stage terminated, some required files were not created."
}

Add any additional information or context about the problem here.

No response

fullcream9 avatar Nov 14 '23 14:11 fullcream9

Hi, @fullcream9. How many of the 400 subjects have this error? It looks like this could be an error with the field maps and/or field map jsons. Does this subject have field maps and if so, do the field map jsons contain intended for fields that match the field maps with the appropriate functional run files? Are the fmap or func files/jsons different somehow from the subjects you've been able to process successfully?

LuciMoore avatar Nov 21 '23 01:11 LuciMoore

Hi,

Thank you for your response and suggestions. I've carefully checked the fmap and func files/jsons for the failed subjects and found no difference between successful subjects.

So far, I have processed 400 subjects in sequence and successfully completed 280 of them before encountering the error. So, about 120 subjects have this error.

To pinpoint the problem, I attempted to process one subject, sub-NDARRR570MC0, individually using the following command:

docker run --rm -v /home/panshi/ABCD_Results/panshi/HBN/ABCD_HCP_pipe/ABCD-BIDS/CDODD_BIDS_raw:/bids_input:ro -v /home/panshi/Documents/ABCD-BIDS/HBN_deri_temp/:/output -v /home/panshi/Downloads/freesurfer_license/:/license dcanumn/abcd-hcp-pipeline /bids_input /output --freesurfer-license=/license/license.txt --ncpus 16 --participant-label NDARRR570MC0 >> NDARRR570MC0_test_newserver.log 2>&1

It appears that the issue arises during the FreeSurfer recon-all step (mris_fix_topolo, in particular). I'm attaching the NDARRR570MC0_test_newserver.log and recon-all.err files for your reference and further insight.

Any additional guidance or insights you could provide would be greatly appreciated.

NDARRR570MC0_test_newserver.log:

    The pipeline must choose distortion correction method based on the
    type(s) of field maps available. The type of fieldmaps you have are
    either not able to be used in the abcd-hcp-pipeline or they are
    not properly identified in the BIDS format. The pipeline does not
    account for 'phasediff', 'magnitude', and 'fieldmap' field maps
    filetypes yet. If you have 'phasediff' and 'magnitude' field maps,
    please provide the original 'phase1', 'phase2', 'magnitude1', and
    'magnitude2' field maps used to calculate those files. The pipeline
    does the calculation itself.

abcd-hcp-pipeline v0.1.3 running PreFreeSurfer /opt/pipeline/PreFreeSurfer/PreFreeSurferPipeline.sh
--path=/output/sub-NDARRR570MC0/ses-None/files
--subject=NDARRR570MC0
--t1=/bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-HCP_T1w.nii.gz@/bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-VNav_T1w.nii.gz@/bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-VNavNorm_T1w.nii.gz
--t2=/bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-VNav_T2w.nii.gz@/bids_input/sub-NDARRR570MC0/anat/sub-NDARRR570MC0_acq-VNavNorm_T2w.nii.gz
--t1template=/opt/pipeline/global/templates/MNI152_T1_1mm.nii.gz
--t1templatebrain=/opt/pipeline/global/templates/MNI152_T1_1mm_brain.nii.gz
--t1template2mm=/opt/pipeline/global/templates/MNI152_T1_2mm.nii.gz
--t2template=/opt/pipeline/global/templates/MNI152_T2_1mm.nii.gz
--t2templatebrain=/opt/pipeline/global/templates/MNI152_T2_1mm_brain.nii.gz
--t2template2mm=/opt/pipeline/global/templates/MNI152_T2_2mm.nii.gz
--templatemask=/opt/pipeline/global/templates/MNI152_T1_1mm_brain_mask.nii.gz
--template2mmmask=/opt/pipeline/global/templates/MNI152_T1_2mm_brain_mask_dil.nii.gz
--brainsize=150
--fnirtconfig=/opt/pipeline/global/config/T1_2_MNI152_2mm.cnf
--fmapmag=NONE
--fmapphase=NONE
--fmapgeneralelectric=NONE
--echodiff=NONE
--SEPhaseNeg=NONE
--SEPhasePos=NONE
--echospacing=NONE
--seunwarpdir=NONE
--t1samplespacing=0.000012019
--t2samplespacing=0.000008138
--unwarpdir=z
--gdcoeffs=NONE
--avgrdcmethod=NONE
--topupconfig=/opt/pipeline/global/config/b02b0.cnf
--useT2=true
--printcom=
--useStudyTemplate=false
--StudyTemplate=NONE
--StudyTemplateBrain=NONE abcd-hcp-pipeline v0.1.3 running FreeSurfer /opt/pipeline/FreeSurfer/FreeSurferPipeline.sh
--subject=NDARRR570MC0
--subjectDIR=/output/sub-NDARRR570MC0/ses-None/files/T1w
--t1=/output/sub-NDARRR570MC0/ses-None/files/T1w/T1w_acpc_dc_restore.nii.gz
--t1brain=/output/sub-NDARRR570MC0/ses-None/files/T1w/T1w_acpc_dc_restore_brain.nii.gz
--t2=/output/sub-NDARRR570MC0/ses-None/files/T1w/T2w_acpc_dc_restore.nii.gz
--useT2=true
--printcom= missing expected outputs from FreeSurfer file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/T1w_hires.nii.gz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/lh.EC_average file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/rh.EC_average file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/lh.aparc.a2009s.annot file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/lh.aparc.annot file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/label/rh.aparc.annot file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/T1w_hires.masked.mgz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aparc+aseg.mgz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aparc.a2009s+aseg.mgz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aseg.hires.mgz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/aseg.hires.nii.gz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/brain.hires.mgz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/lh.ribbon.mgz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/rh.ribbon.mgz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/ribbon.mgz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/wm.hires.mgz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/mri/wmparc.mgz file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/aseg.stats file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.aparc.a2009s.stats file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.aparc.stats file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/lh.curv.stats file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.aparc.a2009s.stats file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.aparc.stats file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/stats/rh.curv.stats file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.mid file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.pial file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.area.prehires file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv.pial file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.curv.prehires file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.pial file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sphere file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sphere.reg file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.sulc file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.thickness file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.volume file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white.deformed file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/lh.white.prehires file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area.mid file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.area.pial file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.curv file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.curv.pial file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.inflated file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.orig file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.pial file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sphere file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sphere.reg file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.sulc file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.thickness file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.volume file not found: /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/surf/rh.white Traceback (most recent call last): File "/app/run.py", line 397, in _cli() File "/app/run.py", line 69, in _cli return interface(**kwargs) File "/app/run.py", line 393, in interface stage.run(ncpus) File "/app/pipelines.py", line 592, in run self.teardown(result) File "/app/pipelines.py", line 538, in teardown self.class.name) Exception: error caught during stage: FreeSurfer

recon-all.err：

SUBJECT NDARRR570MC0 DATE Fri Dec 15 03:11:12 UTC 2023 USER root HOST 569e2b72c878 PROCESSOR x86_64 OS Linux Linux 569e2b72c878 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 2 18:01:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux $Id: recon-all,v 1.379.2.73 2013/05/12 23:15:37 nicks Exp $ /opt/freesurfer/bin/recon-all PWD /output/sub-NDARRR570MC0/ses-None/files/T1w/NDARRR570MC0/scripts CMD mris_fix_topology -mgz -sphere qsphere.nofix -ga -seed 1234 NDARRR570MC0 lh

fullcream9 avatar Dec 15 '23 14:12 fullcream9