nnUNet icon indicating copy to clipboard operation
nnUNet copied to clipboard

Questions about "This output is too large for python process-process communication" during the inference stage.

Open Luoxd1996 opened this issue 3 years ago • 2 comments

Hi Fabian, I meet a problem when using nnUNet for inference, the log print this sentence "This output is too large for python process-process communication. Saving output temporarily to disk", but there are not any prediction output (some temporarily npy files were generated and deleted later). My task is about abdominal multi-organ segmentation, the average volume shape is 220 x 512 x 512. Could you give me some suggestions? Best, Xiangde.

Luoxd1996 avatar May 08 '21 01:05 Luoxd1996

This is just an information. Not a problem. No need to do anything. usually. Have you checked the RAM of your system? Is it getting full? Have you checked the entire text ourput for errors?

FabianIsensee avatar May 08 '21 05:05 FabianIsensee

Hi, I have the same issue as well. There is no prediction output saved in the output folder. How to solve the problem?

dorltcheng avatar Aug 10 '22 13:08 dorltcheng

INFO: Prediction is too large for python process-process communication. Saving to file... It's stuck here while you train the prediction part of the model

Zhangsir125 avatar Jul 21 '23 05:07 Zhangsir125

Hello, I seem to have the same issue. Has anyone managed to solve this issue? If so, how did you do it? Thank you in advance for your help!

t21leluy avatar Aug 28 '23 10:08 t21leluy

Please update nnunet, the problem should no longer exist. Let us know if the issue still persists.

Kobalt93 avatar Sep 20 '23 08:09 Kobalt93

INFO: Prediction is too large for python process-process communication. Saving to file... It's stuck here while you train the prediction part of the model

And now I also have this problem. Can someone help me? I always get stuck on the last predication, and then my validation file never has summary.json.

SuChenHong avatar Apr 13 '24 04:04 SuChenHong

INFO: Prediction is too large for python process-process communication. Saving to file... It's stuck here while you train the prediction part of the model

And now I also have this problem. Can someone help me? I always get stuck on the last predication, and then my validation file never has summary.json.

Did you resolve this problem? I discovered that two samples takes more than an hour to produce predicted results when the inputs are large. The GPU utilization is low. Hope who will give a better solution to it.

angleboy8 avatar May 16 '24 08:05 angleboy8

I have actually the same problem; either none of the expected predictions is saved or only very few of them are. No specific error messages are given in the log files.

bghattas avatar May 18 '24 08:05 bghattas

Hi, I have fixed this issue by increasing the memory size or swapping memory (I tested the 24010241024, and it required more than 150G). Best.

Badih Ghattas @.***> 于2024年5月18日周六 16:39写道:

I have actually the same problem; either none of the expected predictions is saved or only very few of them are. No specific error messages are given in the log files.

— Reply to this email directly, view it on GitHub https://github.com/MIC-DKFZ/nnUNet/issues/629#issuecomment-2118703576, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEV7K52Q4MRNS373EDAZ3GTZC4HS7AVCNFSM44L7L572U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJRHA3TAMZVG43A . You are receiving this because you authored the thread.Message ID: @.***>

Luoxd1996 avatar May 18 '24 11:05 Luoxd1996