23.2.0a3: `template_iterator_wf.select_xfm` failing with `MNIPediatricAsym` when cohort specified

[smeisler]

What happened?

select_xfm is failing on

What command did you use?

singularity run -e --containall -B ${scratch},${cache_dir}\
${fmriprep_img} $scratch/data $scratch/data/derivatives/fmriprep participant \
--participant_label ${subject:4} \
-w $scratch \
--fs-license-file ${scratch}/data/code/license.txt \
--fs-subjects-dir $scratch/data/derivatives/freesurfer/ \
--cifti-output 91k \
--skip-bids-validation \
--stop-on-first-crash \
--nprocs $nprocs \
--omp-nthreads $omp \
--notrack \
--mem-mb ${mem_mb} \
--output-spaces MNIPediatricAsym:cohort-4:res-2 \
--fd-spike-threshold 0.3 --dvars-spike-threshold 0.5 \
--longitudinal

What version of fMRIPrep are you running?

23.2.0a3

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

No

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

Node select_xfm.a0 failed to run on host node101.
231215-01:23:22,460 nipype.workflow ERROR:
	 Saving crash info to /om2/scratch/tmp/oozernov/READER_fitlins/sub-reader403/data/derivatives/fmriprep/sub-reader403/log/20231214-180959_e0b6ad78-2005-43ba-be80-e8a7f8f4d25e/crash-20231215-012322-oozernov-select_xfm.a0-dc6b4a8f-510b-48f6-ae9a-5aa433545778.txt
Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node select_xfm.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 429, in aggregate_outputs
	    predicted_outputs = self._list_outputs()  # Predictions from _list_outputs
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/utility.py", line 217, in _list_outputs
	    index = self.inputs.keys.index(self.inputs.key)
	ValueError: 'MNIPediatricAsym:cohort-4' is not in list


231215-01:23:22,469 nipype.workflow CRITICAL:
	 fMRIPrep failed: Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node select_xfm.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 429, in aggregate_outputs
	    predicted_outputs = self._list_outputs()  # Predictions from _list_outputs
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/utility.py", line 217, in _list_outputs
	    index = self.inputs.keys.index(self.inputs.key)
	ValueError: 'MNIPediatricAsym:cohort-4' is not in list

Additional information / screenshots

BIDS Validator outputs below. I don't think any of these errors would be related to the present error.

[email protected]
(node:34612) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
	1: [ERR] Invalid JSON file. The file is not formatted according the schema. (code: 55 - JSON_SCHEMA_VALIDATION_ERROR)
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-listening_run-1_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-listening_run-2_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-listening_run-3_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-reading_run-1_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-reading_run-2_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-rest_run-1_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./task-listening_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./task-reading_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./task-rest_bold.json
			Evidence: .CogAtlasID should match format "uri"

	Please visit https://neurostars.org/search?q=JSON_SCHEMA_VALIDATION_ERROR for existing conversations about this issue.

	2: [ERR] The value of (EchoTime2 - EchoTime1) should be within the range of 0.0001 - 0.01. (code: 83 - ECHOTIME1_2_DIFFERENCE_UNREASONABLE)
		./sub-reader031/ses-pre/fmap/sub-reader031_ses-pre_acq-a_run-2_phasediff.nii.gz

	Please visit https://neurostars.org/search?q=ECHOTIME1_2_DIFFERENCE_UNREASONABLE for existing conversations about this issue.

	1: [WARN] Tabular file contains custom columns not described in a data dictionary (code: 82 - CUSTOM_COLUMN_WITHOUT_DESCRIPTION)
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-rest_run-1_events.tsv
			Evidence: Columns: TODO -- fill in rows and add more tab-separated columns if desired not defined, please define in: /events.json, /task-rest_events.json,/run-1_events.json,/task-rest_run-1_events.json,/sub-reader031/sub-reader031_events.json,/sub-reader031/sub-reader031_task-rest_events.json,/sub-reader031/sub-reader031_run-1_events.json,/sub-reader031/sub-reader031_task-rest_run-1_events.json,/sub-reader031/ses-pre/sub-reader031_ses-pre_events.json,/sub-reader031/ses-pre/sub-reader031_ses-pre_task-rest_events.json,/sub-reader031/ses-pre/sub-reader031_ses-pre_run-1_events.json,/sub-reader031/ses-pre/sub-reader031_ses-pre_task-rest_run-1_events.json,/sub-reader031/ses-pre/func/sub-reader031_ses-pre_events.json,/sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-rest_events.json,/sub-reader031/ses-pre/func/sub-reader031_ses-pre_run-1_events.json,/sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-rest_run-1_events.json
		./sub-reader031/ses-pre/sub-reader031_ses-pre_scans.tsv
			Evidence: Columns: operator, randstr not defined, please define in: /scans.json, /sub-reader031/sub-reader031_scans.json,/sub-reader031/ses-pre/sub-reader031_ses-pre_scans.json

	Please visit https://neurostars.org/search?q=CUSTOM_COLUMN_WITHOUT_DESCRIPTION for existing conversations about this issue.

	2: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.

        Summary:                  Available Tasks:                          Available Modalities: 
        53 Files, 229.24MB        listening                                 MRI                   
        1 - Subject               reading                                                         
        1 - Session               rest                                                            
                                  TODO: full task name for reading                                
                                  TODO: full task name for listening                              
                                  TODO: full task name for rest                                   


	If you have any questions, please post on https://neurostars.org/tags/bids.

[effigies]

Interesting. Seems to be the cohort/res combo. Can you share sub_reader403_wf/template_iterator_wf/select_xfm/_report/report.rst?

[smeisler]

Node: sub_reader403_wf (template_iterator_wf (select_xfm (utility)
==================================================================


 Hierarchy : fmriprep_23_2_wf.sub_reader403_wf.template_iterator_wf.select_xfm
 Exec ID : select_xfm.a0


Original Inputs
---------------


* anat2std_xfm : ['/om2/scratch/tmp/oozernov/READER_fitlins/sub-reader403/data/derivatives/fmriprep/sub-reader403/anat/sub-reader403_run-1_from-T1w_to-MNIPediatricAsym_mode-image_xfm.h5', '/om2/scratch/tmp/oozernov/READER_fitlins/sub-reader403/data/derivatives/fmriprep/sub-reader403/anat/sub-reader403_run-1_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5', '/om2/scratch/tmp/oozernov/READER_fitlins/sub-reader403/data/derivatives/fmriprep/sub-reader403/anat/sub-reader403_run-1_from-T1w_to-MNI152NLin6Asym_mode-image_xfm.h5']
* key : MNIPediatricAsym:cohort-4
* keys : ['MNIPediatricAsym', 'MNI152NLin2009cAsym', 'MNI152NLin6Asym']

[effigies]

I see. So the issue is that we currently pass around template as if it's meaningful on its own, because resolutions are not considered distinct templates (you can register to 1 and resample to 2). But cohorts should be considered distinct templates, so we need to pass around cohort information.

Right now we pass template and template_spec out of register_template_wf:

https://github.com/nipreps/smriprep/blob/8830bf86c7ea9882b48a6d2de35c160dd342c921/smriprep/workflows/fit/registration.py#L205-L208

And this needs to be combined into {template}[:cohort-<cohort>]. Either that can be done in that workflow or in the fit workflow

https://github.com/nipreps/smriprep/blob/8830bf86c7ea9882b48a6d2de35c160dd342c921/smriprep/workflows/anatomical.py#L971-L1013

Another approach is to pass around template_spec everywhere we pass around template and decide what to do with the collection of keys.

One thing that seems clear to me is that we'll need to add cohort-<label> to the transform file, so that we can distinguish pre-computed transforms.

That will require updates to the path patterns in niworkflows: https://github.com/nipreps/niworkflows/blob/a0af2636521580d84b6b4eb1c32b0cfc4e1bff85/niworkflows/data/nipreps.json#L154-L184

As well as adding the cohort to the how we look up precomputed transforms in:

https://github.com/nipreps/smriprep/blob/8830bf86c7ea9882b48a6d2de35c160dd342c921/smriprep/utils/bids.py#L53-L63

Which is called here:

https://github.com/nipreps/smriprep/blob/8830bf86c7ea9882b48a6d2de35c160dd342c921/smriprep/workflows/base.py#L365-L369

If any changes are needed to the last part, they'll need to be propagated to fMRIPrep here:

https://github.com/nipreps/fmriprep/blob/7f4b302ab916de4d6257cd76f9eba7ef0931c948/fmriprep/workflows/base.py#L250-L263

[effigies]

@smeisler Do you have any time or interest in working into this? Would you need help getting set up with a dev environment? I'm working on another issue today, so I would be glad of any effort you're able to put at this.

cc also @mgxd as this will affect nibabies significantly.

[smeisler]

I can give it a shot. I shouldn't have trouble setting up a dev environment (changes will be happening in https://github.com/smeisler/smriprep/tree/fix/cohort_res) but will update here if I run into issues.

[smeisler]

Actually before I do too much, I should make sure the dev environment was set up correctly. Steps I took:

1. Fork the repo smeisler/smriprep
2. Start a new branch smeisler/smriprep/fix/cohort_res
3. Clone it to machine
4. Switch to my working branch:cd smriprep, git checkout fix/cohort_res
5. Create new conda environement mamba create -n smriprep_dev pip; mamba activate smriprep_dev
6. Install smriprep in that environment from the cloned repo pip install -e .

Anything I'm missing? I realize it's not containerized, so there might be some oddities with software versions I have on my computer, but hopefully I can at least make the changes to the workflow.

[effigies]

You're going to need to work with niworkflows and you'll at least want to test with fmriprep, so I think you'll want to install all of these:

for REPO in nipreps/{fmriprep,smriprep,niworkflows}; do
    git clone https://github.com/$REPO
done
mamba env create -f fmriprep/env.yml  # Can use -n to override the default name of fmriprep
mamba env config vars set -n fmriprep FSLDIR=$CONDA_PREFIX  # Make sure FSL tools can find each other
mamba activate fmriprep
pip install --no-deps -e ./fmriprep -e ./smriprep -e ./niworkflows

It's not hard to add additional repos after the fact.

You'll also need FreeSurfer, AFNI and Connectome Workbench installed in your PATH, but otherwise you should be good to go.

Edit: Note that I reordered one command. You want to set the environment variable before activating to make sure it gets loaded into your shell.

[effigies]

@smeisler Just checking in about how this is going for you.

[smeisler]

Haven't had much time to work on this yet, hope to get to it tomorrow.

[miltoncamachocamacho]

Hello @smeisler have you had any luck? I am facing the same issue but using MNIInfant:cohort-2. ValueError: 'MNIInfant:cohort-2' is not in list

[WangYunHong98]

Hi @effigies

I have the same issue: ValueError: 'MNIPediatricAsym:cohort-1' is not in list. Has this issue been solved or can I use other versions released before fmriprep 23.2.1.

Best, Yunhong

[fphsFischmeister]

Hi @effigies, hi @smeisler, is there any progress on that issue or workaround? We have the same issues with ValueError: 'MNIPediatricAsym:cohort-6' is not in list but would like to proceed with fmriprep 23.2 instead switching to an older version. Thanks in advance, Florian

[effigies]

If you use the :unstable tag, you can test the next version prior to release.

[fphsFischmeister]

Hi @effigies, I tested the current _: unstable _ version with _ MNIPediatricAsym: res-2:cohert-6 _. Everything worked fine for me and did not produce any errors, so I'm looking forward to the 22.2.3 release.

Thanks for all the effort,

Florian

[effigies]

The release that will be included in will be 24.0.0.