smriprep
smriprep copied to clipboard
nipreps
Check continuity of fast-track, esp. multi-session datasets and w/cohort normalization
#219 and #223 will likely break some parts of the anatomical fast track. Let's make sure it works before cutting 0.7.
Hello,
Fast-track seems to break with my dataset on smriprep 0.7 / fmriprep 20.2.0, notably due to bids' build_path not considering potential session/runs within the name.
I developed a dirty-hack which rely on BIDSLayout instead and works for me.
commit: https://github.com/nuKs/smriprep/commit/bd72f83975436cbc0a230570d77e3876a355c121 branch: https://github.com/nuKs/smriprep/commits/nuKs/fix-fast-track
I did not PR as I am not sure it's the best way to fix this, and if this works on other use cases. Notably, I didn't change anything related to Freesurfer files as I didn't have preprocessed sample files. It's tested on a single subject/session dataset, I still need to try it out on multi session dataset as well.
The included logging of missing files is convenient in any case.
The patch also fixes anats within two sessions. The use case appears to be different as for single session anat, no template has to be created, while 2 sessions are averaged. In both case, "run-" is kept.
Here are the outputs for anat derivatives (smriprep) of subsets of the day2day dataset,
with single session:
└── smriprep
├── dataset_description.json
├── logs
│ ├── CITATION.bib
│ ├── CITATION.html
│ ├── CITATION.md
│ └── CITATION.tex
├── sub-01
│ ├── figures
│ │ ├── sub-01_desc-about_T1w.html
│ │ ├── sub-01_desc-summary_T1w.html
│ │ ├── sub-01_ses-01_run-1_desc-conform_T1w.html
│ │ ├── sub-01_ses-01_run-1_dseg.svg
│ │ ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_T1w.svg
│ │ ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_T1w.svg
│ │ └── sub-01_ses-01_run-1_space-OASIS30ANTs_T1w.svg
│ └── ses-01
│ └── anat
│ ├── sub-01_ses-01_run-1_desc-brain_mask.json
│ ├── sub-01_ses-01_run-1_desc-brain_mask.nii.gz
│ ├── sub-01_ses-01_run-1_desc-preproc_T1w.json
│ ├── sub-01_ses-01_run-1_desc-preproc_T1w.nii.gz
│ ├── sub-01_ses-01_run-1_dseg.nii.gz
│ ├── sub-01_ses-01_run-1_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
│ ├── sub-01_ses-01_run-1_from-MNI152NLin6Asym_to-T1w_mode-image_xfm.h5
│ ├── sub-01_ses-01_run-1_from-OASIS30ANTs_to-T1w_mode-image_xfm.h5
│ ├── sub-01_ses-01_run-1_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
│ ├── sub-01_ses-01_run-1_from-T1w_to-MNI152NLin6Asym_mode-image_xfm.h5
│ ├── sub-01_ses-01_run-1_from-T1w_to-OASIS30ANTs_mode-image_xfm.h5
│ ├── sub-01_ses-01_run-1_label-CSF_probseg.nii.gz
│ ├── sub-01_ses-01_run-1_label-GM_probseg.nii.gz
│ ├── sub-01_ses-01_run-1_label-WM_probseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_desc-brain_mask.json
│ ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_desc-preproc_T1w.json
│ ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_dseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_desc-brain_mask.json
│ ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_desc-brain_mask.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_desc-preproc_T1w.json
│ ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_desc-preproc_T1w.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_dseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_label-CSF_probseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_label-GM_probseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-MNI152NLin6Asym_label-WM_probseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-OASIS30ANTs_desc-brain_mask.json
│ ├── sub-01_ses-01_run-1_space-OASIS30ANTs_desc-brain_mask.nii.gz
│ ├── sub-01_ses-01_run-1_space-OASIS30ANTs_desc-preproc_T1w.json
│ ├── sub-01_ses-01_run-1_space-OASIS30ANTs_desc-preproc_T1w.nii.gz
│ ├── sub-01_ses-01_run-1_space-OASIS30ANTs_dseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-OASIS30ANTs_label-CSF_probseg.nii.gz
│ ├── sub-01_ses-01_run-1_space-OASIS30ANTs_label-GM_probseg.nii.gz
│ └── sub-01_ses-01_run-1_space-OASIS30ANTs_label-WM_probseg.nii.gz
└── sub-01.html
double session:
└── smriprep
├── dataset_description.json
├── logs
│ ├── CITATION.bib
│ ├── CITATION.html
│ ├── CITATION.md
│ └── CITATION.tex
├── sub-01
│ ├── anat
│ │ ├── sub-01_run-1_desc-brain_mask.json
│ │ ├── sub-01_run-1_desc-brain_mask.nii.gz
│ │ ├── sub-01_run-1_desc-preproc_T1w.json
│ │ ├── sub-01_run-1_desc-preproc_T1w.nii.gz
│ │ ├── sub-01_run-1_dseg.nii.gz
│ │ ├── sub-01_run-1_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
│ │ ├── sub-01_run-1_from-MNI152NLin6Asym_to-T1w_mode-image_xfm.h5
│ │ ├── sub-01_run-1_from-OASIS30ANTs_to-T1w_mode-image_xfm.h5
│ │ ├── sub-01_run-1_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
│ │ ├── sub-01_run-1_from-T1w_to-MNI152NLin6Asym_mode-image_xfm.h5
│ │ ├── sub-01_run-1_from-T1w_to-OASIS30ANTs_mode-image_xfm.h5
│ │ ├── sub-01_run-1_label-CSF_probseg.nii.gz
│ │ ├── sub-01_run-1_label-GM_probseg.nii.gz
│ │ ├── sub-01_run-1_label-WM_probseg.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin2009cAsym_desc-brain_mask.json
│ │ ├── sub-01_run-1_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin2009cAsym_desc-preproc_T1w.json
│ │ ├── sub-01_run-1_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin2009cAsym_dseg.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin6Asym_desc-brain_mask.json
│ │ ├── sub-01_run-1_space-MNI152NLin6Asym_desc-brain_mask.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin6Asym_desc-preproc_T1w.json
│ │ ├── sub-01_run-1_space-MNI152NLin6Asym_desc-preproc_T1w.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin6Asym_dseg.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin6Asym_label-CSF_probseg.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin6Asym_label-GM_probseg.nii.gz
│ │ ├── sub-01_run-1_space-MNI152NLin6Asym_label-WM_probseg.nii.gz
│ │ ├── sub-01_run-1_space-OASIS30ANTs_desc-brain_mask.json
│ │ ├── sub-01_run-1_space-OASIS30ANTs_desc-brain_mask.nii.gz
│ │ ├── sub-01_run-1_space-OASIS30ANTs_desc-preproc_T1w.json
│ │ ├── sub-01_run-1_space-OASIS30ANTs_desc-preproc_T1w.nii.gz
│ │ ├── sub-01_run-1_space-OASIS30ANTs_dseg.nii.gz
│ │ ├── sub-01_run-1_space-OASIS30ANTs_label-CSF_probseg.nii.gz
│ │ ├── sub-01_run-1_space-OASIS30ANTs_label-GM_probseg.nii.gz
│ │ └── sub-01_run-1_space-OASIS30ANTs_label-WM_probseg.nii.gz
│ ├── figures
│ │ ├── sub-01_desc-about_T1w.html
│ │ ├── sub-01_desc-summary_T1w.html
│ │ ├── sub-01_run-1_desc-conform_T1w.html
│ │ ├── sub-01_run-1_dseg.svg
│ │ ├── sub-01_run-1_space-MNI152NLin2009cAsym_T1w.svg
│ │ ├── sub-01_run-1_space-MNI152NLin6Asym_T1w.svg
│ │ └── sub-01_run-1_space-OASIS30ANTs_T1w.svg
│ ├── ses-01
│ │ └── anat
│ │ └── sub-01_ses-01_run-1_from-orig_to-T1w_mode-image_xfm.txt
│ └── ses-02
│ └── anat
│ └── sub-01_ses-02_run-1_from-orig_to-T1w_mode-image_xfm.txt
└── sub-01.html
@oesteban what are your thoughts on this? Do you have had any feedback about fast-track since the update? I could be missing something as well.
edit: I have the same issue with the IBC dataset with smriprep 0.7.1 and fmriprep 20.2.1, some subjects have 2 anats so it is averaged and thus are working fine (these folders exist: /sub-xx/anat + 2x /sub-xx/ses-yy/anat) while some subjects have only 1 anat and fail (only this folder exists: /sub-xx/ses-yy/anat, /sub-xx/anat is missing).
My patch seems to work, though not up to date with the fmriprep trunk (no config python file anymore etc).
I did not compare my patch output with traditional fmriprep non-fast track output so i am not 100% sure everything is fine (might pick the wrong template etc).
I can integrate it with the latest version if you think it's OK.
edit: ~~actually seems not to work on subject with single anat session.~~ (nevermind, was actually related to how I extract the session)
Hi @nuKs - thanks for your patience, we are seriously limited in time lately - What would you need from us to push this forward? Just check out your branch and test on our settings?
Do you have examples that make the fast-track fail?
Hi, no worries, thanks for sharing your work!
What my patch does is to rely on pybids layout instead of bids build_path in order to retrieve the fast_track derivatives files. Pybids layout does adapt to any bids-valid naming scheme, while build_path does not, but rather directly assumes the file name.
If you have worked on these bits, a simple quick look at my patch code, or even just the way I've described it, may allow you to see if there is any red flag regarding what I did. I made the assumption that fmriprep must have some additional step that picks up the right derivative within the list, and I didn't check this step's code so I am not 100% sure I patched the right way considering the derivative list returned by the patched function is now less restrictive than it used to be. I am perhaps being overly cautious, If you don't see any reason why the patch would cause issues, then everything should be fine,.
So far, examples that make fast-track fails include:
- subjects with a single T1 (as the path is different since no template is made)
- subjects with T1 containing run-x segments within the name.
In my case, IBC has been failing (without fix) for some subjects, modified/reduced day2day datasets too. IBC openneuro: https://openneuro.org/datasets/ds002685/versions/1.3.1 day2day upon request: https://bmcneurosci.biomedcentral.com/articles/10.1186/s12868-017-0383-y
I'll cleanup/update everything to the latest smriprep version so you can test and merge if you wish to.
Jumping in that old convo: I noticed that the fast-track would fail if you process a single anat in a multi-session dataset (and results is output in a session folder) and then try to use that output as anat fast-track for fMRIPrep. Because collect_derivatives does not look into sessions (even if the pattern would allow it), it will not find the preprocessed data and fMRIPrep. I think going with pybids would indeed provide more flexible search options.
Had made that PR if you want to check in, a bit old though. https://github.com/nipreps/smriprep/pull/272