nipreps
Future Broken Process While Running -- Docker 21.0.1
What happened?
Running with command:
fmriprep-docker bidsdatafolder folders/derivatives participant --participant-label 102 --use-aroma --fs-license-file pathtolicense.txt
Just updated with 21.0.1 because I was getting errors with the smooth node for ICA Aroma. Was hoping this would fix it, but am now getting new errors during the middle of it:
220127-22:00:46,986 nipype.workflow INFO: [Node] Finished "fixhdr_skullstrip2", elapsed time 0.259082s. 220127-22:00:48,702 nipype.workflow INFO: [Node] Setting-up "fmriprep_wf.single_subject_102_wf.func_preproc_ses_03_task_rest_wf.bold_std_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.combine_masks" in "/tmp/work/fmriprep_wf/single_subject_102_wf/func_preproc_ses_03_task_rest_wf/bold_std_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/_std_target_MNI152NLin2009cAsym.resnative/combine_masks". exception calling callback for <Future at 0x7f4dd8237b70 state=finished raised BrokenProcessPool> Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks callback(self) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback result = args.result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. exception calling callback for <Future at 0x7f4dd808a208 state=finished raised BrokenProcessPool> Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks callback(self) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback result = args.result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks callback(self) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback result = args.result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. exception calling callback for <Future at 0x7f4dd8171a58 state=finished raised BrokenProcessPool> Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks callback(self) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback result = args.result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks callback(self) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback result = args.result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks callback(self) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback result = args.result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
What command did you use?
fmriprep-docker bidsdatafolder folders/derivatives participant --participant-label 102 --use-aroma --fs-license-file pathtolicense.txt
What version of fMRIPrep are you running?
21.0.1
How are you running fMRIPrep?
Docker
Is your data BIDS valid?
Yes
Are you reusing any previously computed results?
No
Please copy and paste any relevant log output.
220127-22:00:46,986 nipype.workflow INFO:
[Node] Finished "fixhdr_skullstrip2", elapsed time 0.259082s.
220127-22:00:48,702 nipype.workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_102_wf.func_preproc_ses_03_task_rest_wf.bold_std_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.combine_masks" in "/tmp/work/fmriprep_wf/single_subject_102_wf/func_preproc_ses_03_task_rest_wf/bold_std_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/_std_target_MNI152NLin2009cAsym.resnative/combine_masks".
exception calling callback for <Future at 0x7f4dd8237b70 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x7f4dd808a208 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x7f4dd8171a58 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
callback(self)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
result = args.result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Additional information / screenshots
No response
This is most likely a memory error. How much memory and how many CPU cores are available to the process?
9GB, I can allocate more. Also 8 CPUs and swap 1 GB. Is maybe 12 enough? Computer has 16 total I believe. Not sure given that what the optimal number might be.
Ah, OK, that is good to know. I wish, but I think my computer (Mac) is 16GB of memory. Maybe I'll have to do without aroma then.
On Jan 27 2022, at 4:25 pm, Chris Markiewicz @.***> wrote:
Would 24gb be possible? AROMA is a memory hog. — Reply to this email directly, view it on GitHub (https://github.com/nipreps/fmriprep/issues/2700#issuecomment-1023734164), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AOTL6K63D77FJ6DAZI26CYDUYHH7PANCNFSM5M7IUE2A). Triage notifications on the go with GitHub Mobile for iOS (https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or Android (https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub). You are receiving this because you authored the thread.
You can try 16GB, and setting --n-cpus 1. That will keep multiple processes from competing for limited memory. If you reuse the working directory, then it should start at AROMA.
I will give that a try! I assume that will significantly increase the processing time? But happy to do it if it will make it work.
By reuse the working directory, I assume you mean just don't use a different working directory than what I had already been using? On Jan 27 2022, at 4:39 pm, Chris Markiewicz @.***> wrote:
You can try 16GB, and setting --n-cpus 1. That will keep multiple processes from competing for limited memory. If you reuse the working directory, then it should start at AROMA. — Reply to this email directly, view it on GitHub (https://github.com/nipreps/fmriprep/issues/2700#issuecomment-1023741690), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AOTL6K6AKLK6KV6IXIBTPE3UYHJSRANCNFSM5M7IUE2A). Triage notifications on the go with GitHub Mobile for iOS (https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or Android (https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub). You are receiving this because you authored the thread.
Yes, though you can also kill the job after AROMA is done and resume with more cores.
Thank you so much for the help, quick responses, and a way to get it to work.
I tried with --n-cpus 1, and I appreciate this should work, but found it was taking forever for it to even begin aroma (choosing instead to rerun all previous processes except for recon again). I assume I must be doing some aspect of the command wrong if it is not speeding right to the AROMA section like you said.
I am now trying to --n-cpus 4, and it is still taking forever to reach aroma, but hopefully that will work. I am hoping to find some method to optimize between speed and memory.
Otherwise I will keep chugging along with a lower --n-cpus and just keep it running.
Thanks again!