containers
containers copied to clipboard
ReproNim
Fails while running on OSX
Hi, intrepid user here. Running through the 'walkthrough'; in the 'containers-run' step I get the following error:
(base) kennedyd@MCSMLS065401 ds000003-qc % datalad containers-run
--container-name bids-mriqc
--input sourcedata
--output .
'{inputs}' '{outputs}' participant --participant-label sub-02 -w workdir [INFO ] Making sure inputs are available (this may take some time) get(ok): containers/images/bids/bids-mriqc--0.16.0.sing (file) [from origin...] [INFO ] == Command start (output follows) =====
240523-20:51:23,365 cli IMPORTANT:Running MRIQC version 0.16.0: * BIDS dataset path: /Users/kennedyd/my-experiments/ds000003-qc/sourcedata. * Output folder: /Users/kennedyd/my-experiments/ds000003-qc. * Analysis levels: ['participant'].You are using MRIQC v0.16.0, and a newer version is available: 24.0.0. Traceback (most recent call last): File "/usr/local/miniconda/bin/mriqc", line 10, in
sys.exit(main()) File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/cli/run.py", line 37, in main with Manager() as mgr: File "/usr/local/miniconda/lib/python3.7/multiprocessing/context.py", line 56, in Manager m.start() File "/usr/local/miniconda/lib/python3.7/multiprocessing/managers.py", line 543, in start self._process.start() File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 112, in start self._popen = self._Popen(self) File "/usr/local/miniconda/lib/python3.7/multiprocessing/context.py", line 291, in _Popen return Popen(process_obj) File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_forkserver.py", line 35, in init super().init(process_obj) File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_fork.py", line 20, in init self._launch(process_obj) File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_forkserver.py", line 51, in _launch self.sentinel, w = forkserver.connect_to_new_process(self._fds) File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 64, in connect_to_new_process self.ensure_running() File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 118, in ensure_running listener.bind(address) OSError: [Errno 22] Invalid argument [INFO ] == Command exit (modification check follows) ===== run(error): /Users/kennedyd/my-experiments/ds000003-qc (dataset) [./containers/scripts/singularity_cmd run...] action summary:
get (notneeded: 2, ok: 1) run (error: 1) save (notneeded: 3)
I'm running on a Mac: 14.1 (23B74) Docker: 4.28.0 (139021) Engine: Engine: 25.0.3
FWIW, running the "monolythic" version of walkthrough fails but not like above:
(datalad) yoh@datalads-imac2 containers % bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')
...
> containers/scripts/freeze_versions --save-dataset=. bids-mriqc=0.16.0
I: We will be copying/freezing versions in .
I: bids-mriqc -> 0.16.0
add(ok): .datalad/config (file)
save(ok): . (dataset)
action summary:
add (ok: 1)
save (ok: 1)
> datalad install -d . -s https://github.com/ReproNim/ds000003-demo sourcedata
[INFO ] Remote origin not usable by git-annex; setting annex-ignore
[INFO ] https://github.com/ReproNim/ds000003-demo/config download failed: Not Found
[INFO ] access to 1 dataset sibling s3-PRIVATE not auto-enabled, enable with:
| datalad siblings -d "/private/var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/sourcedata" enable -s s3-PRIVATE
install(ok): sourcedata (dataset)
add(ok): sourcedata (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)
action summary:
add (ok: 3)
install (ok: 1)
save (ok: 2)
> echo workdir/
> datalad save -m 'Ignore workdir' .gitignore
add(ok): .gitignore (file)
save(ok): . (dataset)
action summary:
add (ok: 1)
save (ok: 1)
> datalad containers-run -n bids-mriqc --input sourcedata --output . '{inputs}' '{outputs}' participant group -w workdir
[INFO ] Making sure inputs are available (this may take some time)
get(ok): sourcedata/sub-02/anat/sub-02_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-02/anat/sub-02_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-02/func/sub-02_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/func/sub-13_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): containers/images/bids/bids-mriqc--0.16.0.sing (file) [from origin...]
[INFO ] == Command start (output follows) =====
240529-19:51:23,714 cli IMPORTANT:
Running MRIQC version 0.16.0:
* BIDS dataset path: /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/sourcedata.
* Output folder: /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc.
* Analysis levels: ['group', 'participant'].
You are using MRIQC v0.16.0, and a newer version is available: 24.0.0.
240529-19:51:24,443 nipype.utils WARNING:
A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-19:51:26,343 nipype.utils WARNING:
A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-19:51:37,408 cli WARNING:
IMPORTANT: Anonymized quality metrics (IQMs) will be submitted to MRIQC's metrics repository. Submission of IQMs can be disabled using the ``--no-sub`` argument. Please visit https://mriqc.readthedocs.io/en/latest/dsa.html to revise MRIQC's Data Sharing Agreement.
240529-19:51:37,910 nipype.workflow WARNING:
Some nodes exceed the total amount of memory available (1.75GB).
240529-19:51:39,117 nipype.utils WARNING:
A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-19:51:39,120 nipype.utils WARNING:
A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-19:51:39,135 nipype.utils WARNING:
A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-19:51:39,225 nipype.utils WARNING:
A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.0
240529-20:00:03,917 cli WARNING:
Building bold report: no exclude index was found
240529-20:03:12,53 cli WARNING:
Building bold report: no exclude index was found
The use of ``template_resolution`` is deprecated
The use of ``template_resolution`` is deprecated
The use of ``template_resolution`` is deprecated
240529-20:07:23,275 nipype.workflow WARNING:
Storing result file without outputs
240529-20:07:23,288 nipype.workflow WARNING:
[Node] Error on "mriqc_wf.anatMRIQC.AirMaskWorkflow.invert_xfm" (/var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/workdir/mriqc_wf/anatMRIQC/AirMaskWorkflow/_in_file_..var..folders..tk..5_lnk94x727gwq_x4z6mf1x00000gp..T..repro-8UTZo4N..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/invert_xfm)
240529-20:07:24,497 nipype.workflow ERROR:
Node invert_xfm.a1 failed to run on host 76f636198895.
240529-20:07:24,687 nipype.workflow ERROR:
Saving crash info to /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/logs/crash-20240529-200724-host-user-invert_xfm.a1-9dd79aa3-82a2-4624-a99c-aeca769eed9b.txt
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 434, in run
runtime = self._run_interface(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 829, in _run_interface
self.raise_exception(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 760, in raise_exception
).format(**runtime.dictcopy())
RuntimeError: Command:
antsApplyTransforms --default-value 0 --dimensionality 3 --float 1 --input /home/bidsapp/.cache/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_desc-head_mask.nii.gz --interpolation MultiLabel --output tpl-MNI152NLin2009cAsym_res-01_desc-head_mask_trans.nii.gz --reference-image /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/workdir/mriqc_wf/anatMRIQC/AFNISkullStripWorkflow/_in_file_..var..folders..tk..5_lnk94x727gwq_x4z6mf1x00000gp..T..repro-8UTZo4N..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/binarize/sub-13_T1w_conformed_calc_mask.nii.gz --transform /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/workdir/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..var..folders..tk..5_lnk94x727gwq_x4z6mf1x00000gp..T..repro-8UTZo4N..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/SpatialNormalization/ants_t1_to_mniInverseComposite.h5
Standard output:
Standard error:
Killed
Return code: 137
Traceback (most recent call last):
File "/usr/local/miniconda/bin/mriqc", line 10, in <module>
sys.exit(main())
File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/cli/run.py", line 69, in main
mriqc_wf.run(**config.nipype.get_plugin())
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 632, in run
runner.run(execgraph, updatehash=updatehash, config=self.config)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/base.py", line 164, in run
self._clean_queue(jobid, graph, result=result)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/base.py", line 227, in _clean_queue
raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 434, in run
runtime = self._run_interface(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 829, in _run_interface
self.raise_exception(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 760, in raise_exception
).format(**runtime.dictcopy())
RuntimeError: Command:
antsApplyTransforms --default-value 0 --dimensionality 3 --float 1 --input /home/bidsapp/.cache/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_desc-head_mask.nii.gz --interpolation MultiLabel --output tpl-MNI152NLin2009cAsym_res-01_desc-head_mask_trans.nii.gz --reference-image /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/workdir/mriqc_wf/anatMRIQC/AFNISkullStripWorkflow/_in_file_..var..folders..tk..5_lnk94x727gwq_x4z6mf1x00000gp..T..repro-8UTZo4N..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/binarize/sub-13_T1w_conformed_calc_mask.nii.gz --transform /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc/workdir/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..var..folders..tk..5_lnk94x727gwq_x4z6mf1x00000gp..T..repro-8UTZo4N..ds000003-qc..sourcedata..sub-13..anat..sub-13_T1w.nii.gz/SpatialNormalization/ants_t1_to_mniInverseComposite.h5
Standard output:
Standard error:
Killed
Return code: 137
[INFO ] == Command exit (modification check follows) =====
run(error): /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T/repro-8UTZo4N/ds000003-qc (dataset) [./containers/scripts/singularity_cmd run...]
action summary:
get (notneeded: 2, ok: 7)
run (error: 1)
DataLad 1.0.2 WTF (system)
WTF
system
- distribution: darwin/20.3.0 10.16/x86_64
- encoding:
- default: utf-8
- filesystem: utf-8
- locale.prefered: UTF-8
- filesystem:
- CWD:
- max_pathlength: 1024
- mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
- path: /Users/yoh/proj/repronim/containers
- type: apfs
- HOME:
- max_pathlength: 1024
- mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
- path: /Users/yoh
- type: apfs
- TMP:
- max_pathlength: 1024
- mount_opts: ro,local,rootfs,dovolfs,journaled,multilabel
- path: /var/folders/tk/5_lnk94x727gwq_x4z6mf1x00000gp/T
- type: apfs
- CWD:
- max_path_length: 291
- name: Darwin
- release: 20.3.0
- type: posix
- version: Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64
@djarecka @satra what would happen on your mac if you do
git clone https://github.com/ReproNim/containers
cd containers
bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')
? (@dnkennedy try doing that exact set of commands too for consistency)
did you may be observe that
File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 64, in connect_to_new_process
self.ensure_running()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 118, in ensure_running
listener.bind(address)
OSError: [Errno 22] Invalid argument
before under docker on macs?
Also tested on a mac, and seeing a different issue with multiprocessing
240620-07:50:10,370 cli IMPORTANT:
Running MRIQC version 0.16.0:
* BIDS dataset path: /var/folders/r6/mcdps6rs37x2ttd1fn032yy40000gp/T/repro-AaUAJmz/ds000003-qc/sourcedata.
* Output folder: /var/folders/r6/mcdps6rs37x2ttd1fn032yy40000gp/T/repro-AaUAJmz/ds000003-qc.
* Analysis levels: ['group', 'participant'].
You are using MRIQC v0.16.0, and a newer version is available: 24.0.0.
Traceback (most recent call last):
File "/usr/local/miniconda/bin/mriqc", line 10, in <module>
sys.exit(main())
File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/cli/run.py", line 37, in main
with Manager() as mgr:
File "/usr/local/miniconda/lib/python3.7/multiprocessing/context.py", line 56, in Manager
m.start()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/managers.py", line 543, in start
self._process.start()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 64, in connect_to_new_process
self.ensure_running()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/forkserver.py", line 119, in ensure_running
os.chmod(address, 0o600)
OSError: [Errno 22] Invalid argument: '/tmp/pymp-e12c8p4i/listener-24orxuis'
[INFO ] == Command exit (modification check follows) =====
run(error): /var/folders/r6/mcdps6rs37x2ttd1fn032yy40000gp/T/repro-AaUAJmz/ds000003-qc (dataset) [./code/containers/scripts/singularity_cm...]
action summary:
get (notneeded: 2, ok: 7)
run (error: 1)
I vaguely remember a multiprocessing bug in 3.7 but cannot quickly find anything - @oesteban ring any bells?
Yarik hoped the new version would work. It didn't.
output of running
% datalad containers-run \
-n bids-mriqc \
--input sourcedata \
--output . \
'{inputs}' '{outputs}' participant group -w workdir
[INFO ] Making sure inputs are available (this may take some time)
get(ok): sourcedata/sub-02/anat/sub-02_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-02/anat/sub-02_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-02/func/sub-02_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/func/sub-13_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): code/containers/images/bids/bids-mriqc--24.0.0.sing (file) [from origin...]
[INFO ] == Command start (output follows) =====
2024-06-21 05:06:43 | WARNING | matplotlib | Matplotlib created a temporary cache directory at /tmp/matplotlib-pmaf7zp9 because the default path (/home/mriqc/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
------------------------------------------------------------------
Running MRIQC version 24.1.0.dev0+g3fe90466.d20240417
----------------------------------------------------------------
* BIDS dataset path: /Users/kennedyd/my-experiments/ds000003-qc/sourcedata.
* Output folder: ..
* Analysis levels: ['participant', 'group'].
------------------------------------------------------------------
Process SyncManager-1:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.11/multiprocessing/managers.py", line 592, in _run_server
server = cls._Server(registry, address, authkey, serializer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/multiprocessing/managers.py", line 156, in __init__
self.listener = Listener(address=address, backlog=128)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/multiprocessing/connection.py", line 464, in __init__
self._listener = SocketListener(address, family, backlog)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/multiprocessing/connection.py", line 607, in __init__
self._socket.bind(address)
OSError: [Errno 22] Invalid argument
Traceback (most recent call last):
File "/opt/conda/bin/mriqc", line 8, in <module>
sys.exit(main())
^^^^^^
File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 104, in main
with Manager() as mgr:
^^^^^^^^^
File "/opt/conda/lib/python3.11/multiprocessing/context.py", line 57, in Manager
m.start()
File "/opt/conda/lib/python3.11/multiprocessing/managers.py", line 567, in start
self._address = reader.recv()
^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
buf = self._recv(4)
^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/multiprocessing/connection.py", line 399, in _recv
raise EOFError
EOFError
[INFO ] == Command exit (modification check follows) =====
run(error): /Users/kennedyd/my-experiments/ds000003-qc (dataset) [./code/containers/scripts/singularity_cm...]
add(ok): .bids_db/layout_index.sqlite (file)
add(ok): logs/mriqc-20240621-050643_0195cf7d-f0f5-4206-a443-09895e75f460.log (file)
save(ok): . (dataset)
action summary:
add (ok: 2)
get (notneeded: 2, ok: 7)
run (error: 1)
save (notneeded: 2, ok: 1)