mriqc
mriqc copied to clipboard
nipreps
IndexError: list index out of range (MRIQC never runs longer than 25s)
What happened?
Received an 'IndexError: list index out of range' very early on whenever attempting to run MRIQC.
I have a feeling that it has something to do with the afni version. Please see error output and additional information below.
I am able to run fmriprep just fine.
What command did you use?
singularity run --cleanenv -B /data/ncl-mb10:/data/ncl-mb10,/data/ncl-mb13:/data/ncl-mb13,/usr/bin:/usr/bin $sifloc --participant-label $subj --n_procs 4 --omp-nthreads 4 --mem_gb 16 --no-sub --ica --fft-spikes-detector -w $workdir $indir $outdir participant
What version of fMRIPrep are you running?
MRIQC v23.0.0
How are you running this software?
Singularity
Is your data BIDS valid?
Yes
Are you reusing any previously computed results?
No
Please copy and paste any relevant log output.
Process Process-2:
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.9/site-packages/mriqc/cli/workflow.py", line 61, in build_workflow
retval["workflow"] = init_mriqc_wf()
File "/opt/conda/lib/python3.9/site-packages/mriqc/workflows/core.py", line 44, in init_mriqc_wf
workflow.add_nodes([fmri_qc_workflow()])
File "/opt/conda/lib/python3.9/site-packages/mriqc/workflows/functional.py", line 130, in fmri_qc_workflow
iqmswf = compute_iqms()
File "/opt/conda/lib/python3.9/site-packages/mriqc/workflows/functional.py", line 322, in compute_iqms
fwhm_interface = get_fwhmx()
File "/opt/conda/lib/python3.9/site-packages/mriqc/workflows/utils.py", line 178, in get_fwhmx
afni_version = Info.version()
File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 1098, in version
klass._version = klass.parse_version(raw_info)
File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/afni/base.py", line 36, in parse_version
version_stamp = raw_info.split("\n")[0].split("Version ")[1]
IndexError: list index out of range
Additional information / screenshots
I have a feeling that 'IndexError: list index out of range' might be coming from the afni version. If I run this command: singularity exec np_mriqc_latest.sif afni -ver, I get: Precompiled binary linux_openmp_64: Aug 19 2022 (Version AFNI_22.2.07 'Marcus Aurelius')
The result for: version_stamp = raw_info.split("\n")[0].split("Version ")[1] would be AFNI_22.2.07 'Marcus Aurelius')
This may actually be an error with being unable to find/get the proper raw_info. If need be, I am open to going over this on zoom. I can also be contacted here: [email protected] Thank you.
Hi-
Just to note, it might be simpler to get the version information using the -vnum option, rather than -ver.
The output is simpler for the version number, which likely might not even need to be split, for example:
$ afni -vnum
AFNI_23.0.07
$ afni -ver
Precompiled binary macos_10.12_local: Mar 1 2023 (Version AFNI_23.0.07 'Commodus')
--pt
@hannahleechoi , how did you generate this singularity image? I wonder how this issue has not popped up within our integration tests.
@oesteban Sorry for not seeing this earlier! I generated it like this: singularity build np_mriqc_latest.sif docker://nipreps/mriqc:latest
I think this specific issue was resolved by removing the binding of /usr/bin and making sure directories are in BIDS format.
However, I am still unable to run mriqc. You can see the newer issue here with netstat and 3dinfo: https://neurostars.org/t/bin-netstat-command-not-found-3dinfo-command-not-found-running-through-latest-mriqc-sif/25387
It looks again like a conflict of environments, and AFNI's binaries are being shadowed.
@oesteban Is there a possible way to fix this? Whenever I attempt to run a subject through, this is the furthest I get, and I'm not too sure what to try next.
I would try to first run it on some local desktop or laptop through docker. Once you have it working on one or two subjects, then try the setup with singularity.
This documentation -> https://www.nipreps.org/apps/framework/ can be useful. Please note the two sections following that I linked, one about docker and the other about singularity.
Hi-
Just to note, it might be simpler to get the version information using the
-vnumoption, rather than-ver.The output is simpler for the version number, which likely might not even need to be split, for example:
$ afni -vnum AFNI_23.0.07 $ afni -ver Precompiled binary macos_10.12_local: Mar 1 2023 (Version AFNI_23.0.07 'Commodus')--pt
Thanks Paul for the note, definitely worth updating on the nipype side (very dutifully you already submitted a patch :). However, the singularity container should go with an AFNI version old enough not to enter this error condition.
FWIW, I also ran into this error:
Process Process-2:
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/src/mriqc/mriqc/cli/workflow.py", line 61, in build_workflow
retval["workflow"] = init_mriqc_wf()
File "/src/mriqc/mriqc/workflows/core.py", line 44, in init_mriqc_wf
workflow.add_nodes([fmri_qc_workflow()])
File "/src/mriqc/mriqc/workflows/functional/base.py", line 128, in fmri_qc_workflow
iqmswf = compute_iqms()
File "/src/mriqc/mriqc/workflows/functional/base.py", line 300, in compute_iqms
fwhm_interface = get_fwhmx()
File "/src/mriqc/mriqc/workflows/utils.py", line 178, in get_fwhmx
afni_version = Info.version()
File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 1097, in version
klass._version = klass.parse_version(raw_info)
File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/afni/base.py", line 36, in parse_version
version_stamp = raw_info.split("\n")[0].split("Version ")[1]
IndexError: list index out of range
after building a docker image locally from 5bb5947af8e0ad5cfa061baf589793eb4e3dd706 on my M1 mac:
docker build -t eilidhmacnicol/mriqc:dev --platform=linux/amd64 /Users/eilidhmacnicol/projects/mriqc`
docker run --platform=linux/amd64 --rm -it -v /Users/eilidhmacnicol/projects/temp/:/tmp -v /Users/eilidhmacnicol/projects/mriqc/:/src/mriqc -v /Users/eilidhmacnicol/projects/data/:/data --entrypoint /bin/bash eilidhmacnicol/mriqc:dev
checking the afni install gives:
root@d213cdb85692:/tmp# afni_system_check.py -check_all
-------------------------------- general ---------------------------------
architecture: 64bit ELF
cpu type: x86_64
system: Linux
release: 5.15.49-linuxkit
version: #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022
distribution: ubuntu 20.04 focal
number of CPUs: 6
apparent login shell: bash
shell RC file: .bashrc (exists)
--------------------- AFNI and related program tests ---------------------
which afni : /opt/afni/afni
afni version : /opt/afni/afni: error while loading shared libraries: libXp.so.6: cannot open shared object file: No such file or directory
:
AFNI_version.txt : AFNI_23.0.07, linux_openmp_64, Mar 01 2023
which python : /opt/conda/bin/python
python version : 3.9.12
which R :
which tcsh : /usr/bin/tcsh
instances of various programs found in PATH:
afni : 1 (/opt/afni/afni)
R : 0
python : 1 (/opt/conda/bin/python3.9)
python2 : 0
python3 : 2
/opt/conda/bin/python3.9
/usr/bin/python3.8
** have python3 but not python2
testing ability to start various programs...
afni : FAILURE
/opt/afni/afni: error while loading shared libraries: libXp.so.6: cannot open shared object file: No such file or directory
suma : FAILURE
/opt/afni/suma: error while loading shared libraries: libGLU.so.1: cannot open shared object file: No such file or directory
3dSkullStrip : FAILURE
/opt/afni/3dSkullStrip: error while loading shared libraries: libGLU.so.1: cannot open shared object file: No such file or directory
uber_subject.py : success
3dAllineate : FAILURE
/opt/afni/3dAllineate: error while loading shared libraries: libXp.so.6: cannot open shared object file: No such file or directory
3dRSFC : FAILURE
/opt/afni/3dRSFC: error while loading shared libraries: libXp.so.6: cannot open shared object file: No such file or directory
SurfMesh : FAILURE
/opt/afni/SurfMesh: error while loading shared libraries: libGLU.so.1: cannot open shared object file: No such file or directory
3dClustSim : FAILURE
/opt/afni/3dClustSim: error while loading shared libraries: libXp.so.6: cannot open shared object file: No such file or directory
3dMVM : success
checking for R packages...
rPkgsInstall -pkgs ALL -check : FAILURE
R: Command not found.
R RHOME : R: Command not found.
checking for $HOME files...
.afnirc : missing
.sumarc : missing
.afni/help/all_progs.COMP : missing
------------------------------ python libs -------------------------------
** failed to load module PyQt4
-- PyQt4 is no longer needed for an AFNI bootcamp
++ module loaded: matplotlib.pyplot
module file : /opt/conda/lib/python3.9/site-packages/matplotlib/pyplot.py
matplotlib version : 3.5.2
-------------------------------- env vars --------------------------------
PATH = /opt/ants:/opt/afni:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH =
R_LIBS =
LD_LIBRARY_PATH = /usr/lib/x86_64-linux-gnu:/opt/conda/lib
DYLD_LIBRARY_PATH =
DYLD_FALLBACK_LIBRARY_PATH =
----------------------------- eval dot files -----------------------------
-- no .tcshrc, will create one as a follower of .cshrc
-- considered opertaions: path, flatdir, apsearch
(not on a mac, should skip flatdir)
-- note: followers should not need edits, so edit flags should be 0
(have 1 follower(s), which can be ignored)
dot file test : want 4 modifications across 3 files:
file path flatdir apsearch follower
--------------- ---- ------- -------- --------
.cshrc 1 0 1 0
.tcshrc 0 0 0 1
.bashrc 1 0 1 0
------------------------------ data checks -------------------------------
data dir : missing AFNI_data6
data dir : missing AFNI_demos
data dir : missing suma_demo
data dir : missing afni_handouts
atlas : found TT_N27+tlrc under /opt/afni
------------------------------ OS specific -------------------------------
which apt-get : /usr/bin/apt-get
apt-get version : apt 2.0.8 (amd64)
have Ubuntu system: ubuntu 20.04 focal
so despite building with the amd architecture (and throwing no errors during the build), some afni libraries appear to be missing. I also ran this on an intel mac (without the --platform flag) and it ran fine.
This was resolved (for my use case) by pulling the latest docker image rather than building. Hopefully this helps shine some light on what's happening.
@oesteban @eilidhmacnicol Thank you so much for the suggestions. I tried to run the container through Docker with the BIDS-formatted data on my personal computer. I didn't get the netstat and 3dinfo errors, but I did get an error regarding synthstrip. I copied the crash file below. Some extra info: I pulled the image and ran mriqc with the command below:
docker run -it --rm \
-v /Users/hannahleechoi/ncl/SPIN_TEST:/data:ro \
-v /Users/hannahleechoi/ncl/SPIN_TEST/derivatives/mriqc:/out \
nipreps/mriqc:latest \
/data /out participant --participant_label sub-SPIN007
It is my first time working with containers, so I am wondering if you have any general tips on getting things to run smoothly. My assumption is that as long as you have a BIDS-valid dataset, MRIQC should work fine. Is this correct?
Also, the reason I would like to use MRIQC is because it closely mirrors fMRIPrep which I am using for my project and have working smoothly. I am working with clinical data and would like to choose the best T1w and best functional run prior to running fMRIPrep on the subjects. If you have other recommendations or comments on my approach, please let me know.
Node: mriqc_wf.anatMRIQC.synthstrip_wf.synthstrip
Working directory: /tmp/work/mriqc_wf/anatMRIQC/synthstrip_wf/_in_file_..data..sub-SPIN007..anat..sub-SPIN007_T1w.nii.gz/synthstrip
Node inputs:
args = <undefined>
border_mm = 1
environ = {}
in_file = <undefined>
model = /opt/freesurfer/models/synthstrip.1.pt
out_file = <undefined>
out_mask = <undefined>
use_gpu = False
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/mriqc/engine/plugin.py", line 60, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthstrip.
Cmdline:
synthstrip -b 1 -i /tmp/work/mriqc_wf/anatMRIQC/synthstrip_wf/_in_file_..data..sub-SPIN007..anat..sub-SPIN007_T1w.nii.gz/pre_n4/clipped_corrected.nii.gz --model /opt/freesurfer/models/synthstrip.1.pt -o clipped_corrected_desc-brain.nii.gz -m clipped_corrected_desc-brain_mask.nii.gz
Stdout:
Stderr:
Killed
Traceback:
RuntimeError: subprocess exited with code 137.
@hannahleechoi Error code 137 is due to insufficient memory. How much RAM are you devoting to the process (and over how many CPUs)?
@smeisler Oscar suggested trying to get it work on my personal mac first - the error code makes sense. I felt like it was taking a bit too long to get mriqc working for me, so I decided to handle qc through fmriprep. Thanks for checking in on the issue.
@hannahleechoi my 2ct:
- It seems very clear to me that @smeisler's assessment is correct.
- I don't think you really want to replace MRIQC with fMRIPrep - the goals are different. There's more on this in our protocol (OA preprint) and in this piece by Niso et al..
Hi @oesteban -- I'm getting the same IndexError with the latest pre-release for 24.0.0, but maybe it's a conflicting environment issue? I'm using --cleanenv so I thought that option would prevent that, but I am likely misunderstanding something.
Using the same data and code, I'm not getting any errors with 23.1.0. Maybe one notable feature of my dataset is that it is multiecho, and it seems like the handling for multiecho data has improved since 23.1.0. The IndexError appears starting with 23.1.1.
Happy to share data if it's helpful. It's not quite ready for OpenNeuro and public sharing, but maybe I send a few subjects privately via OneDrive or DropBox.
Thanks for any help!
Best wishes, David
Here is my code:
TEMPLATEFLOW_DIR=/ZPOOL/data/tools/templateflow
export APPTAINERENV_TEMPLATEFLOW_HOME=/opt/templateflow
/ZPOOL/data/tools/apptainer/bin/singularity run --cleanenv \
-B ${TEMPLATEFLOW_DIR}:/opt/templateflow \
-B $maindir:/base \
-B $scratchdir:/scratch \
/ZPOOL/data/tools/mriqc-24.0.0rc2.simg \
/base/bids /base/derivatives/mriqc \
participant --participant_label $sub \
--bids-filter-file /base/code/mriqc_config.json \
-w /scratch
Here is the contents of my mriqc_config.json file:
{
"bold": {"datatype": "func", "suffix": "bold", "part": [null, "mag"]}
}
And here is the full output of the error:
------------------------------------------------------------------
Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
----------------------------------------------------------------
* BIDS dataset path: /base/bids.
* Output folder: /base/derivatives/mriqc.
* Analysis levels: ['participant'].
* BIDS filters-file: /base/code/mriqc_config.json.------------------------------------------------------------------
Process Process-2:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/workflow.py", line 45, in build_workflow
config.load(config_file)
File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
settings = loads(filename.read_text())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
ret = decoder.load_line(line, currentlevel, multikey,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
value, vtype = self.load_value(pair[1], strictly_valid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
return (self.load_array(v), "array")
^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
a[b] = a[b] + ',' + a[b + 1]
~^^^^^^^
IndexError: list index out of range
Traceback (most recent call last):
File "/opt/conda/bin/mriqc", line 8, in <module>
sys.exit(main())
^^^^^^
File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 123, in main
config.load(config_file)
File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
settings = loads(filename.read_text())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
ret = decoder.load_line(line, currentlevel, multikey,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
value, vtype = self.load_value(pair[1], strictly_valid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
return (self.load_array(v), "array")
^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
a[b] = a[b] + ',' + a[b + 1]
~^^^^^^^
IndexError: list index out of range
@DVSneuro I think this is not the same IndexError. For some reason, your config file might be corrupted.
I just released 24.0.0rc2 yesterday -- can you give it a try? We have improved the logging a lot, so there will be a log entry at the beginning telling you about this config file path.
Thanks, @oesteban -- sorry, I wasn't sure if I should create a new issue for this error. I do get the same error with the 24.0.0rc2 version. I can also confirm that it happens without the config file being used.
I got a different error with a different subject (see below). So, just as a sanity check, I tripled checked the bids validator, and there are no errors there.
Same error, latest pre-release:
bash mriqc.sh 10003
------------------------------------------------------------------
Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
----------------------------------------------------------------
* BIDS dataset path: /base/bids.
* Output folder: /base/derivatives/mriqc.
* Analysis levels: ['participant'].
* BIDS filters-file: /base/code/mriqc_config.json.------------------------------------------------------------------
Process Process-2:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/workflow.py", line 45, in build_workflow
config.load(config_file)
File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
settings = loads(filename.read_text())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
ret = decoder.load_line(line, currentlevel, multikey,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
value, vtype = self.load_value(pair[1], strictly_valid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
return (self.load_array(v), "array")
^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
a[b] = a[b] + ',' + a[b + 1]
~^^^^^^^
IndexError: list index out of range
Traceback (most recent call last):
File "/opt/conda/bin/mriqc", line 8, in <module>
sys.exit(main())
^^^^^^
File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 123, in main
config.load(config_file)
File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
settings = loads(filename.read_text())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
ret = decoder.load_line(line, currentlevel, multikey,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
value, vtype = self.load_value(pair[1], strictly_valid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
return (self.load_array(v), "array")
^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
a[b] = a[b] + ',' + a[b + 1]
~^^^^^^^
IndexError: list index out of range
Different subject, new error:
------------------------------------------------------------------
Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
----------------------------------------------------------------
* BIDS dataset path: /base/bids.
* Output folder: /base/derivatives/mriqc.
* Analysis levels: ['participant'].
------------------------------------------------------------------
Process Process-2:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
ret = decoder.load_line(line, currentlevel, multikey,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
value, vtype = self.load_value(pair[1], strictly_valid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
return (self.load_array(v), "array")
^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1029, in load_array
raise ValueError("Not a homogeneous array")
ValueError: Not a homogeneous array
@oesteban -- I tried the latest pre-release (rc6), and I'm still getting the same IndexError. I uploaded some of the data to OpenNeuro if you want to take a look (https://openneuro.org/datasets/ds005085). I don't see the error on sub-10006 (no bold scans) or sub-10015 (no multiecho scans), so I wonder if the issue has something to do with how MRIQC is handling multiecho data?
I'm going to close the original issue, as the IndexError is unrelated and it seems to have been solved. Let's follow up with @DVSneuro on #1264.