nnUNet
nnUNet copied to clipboard
MIC-DKFZ
ValueError when running prediction
Hi, as I was running nnUNetv2_predict I encountered the following error and wondered how can I solve it. Is there anyway nnUNetv2_predict can skip error prediction cases? Thank you!
My command:
nnUNetv2_predict -d Dataset220_KiTS2023 -i input_folder -o output_folder -f 0 1 2 3 4 -tr nnUNetTrainer -c 3d_lowres -p nnUNetPlans -npp 1 -nps 1 --continue_prediction
The error:
Predicting case_00676:
perform_everything_on_device: True
0%| | 0/16 [00:00<?, ?it/s]/home/siw669/anaconda3/envs/nnunet/lib/python3.12/site-packages/skimage/transform/_warps.py:160: RuntimeWarning: divide by zero encountered in divide
factors = np.divide(input_shape, output_shape)
100%|███████████████████████████████████████████| 16/16 [00:10<00:00, 1.47it/s]
44%|███████████████████▎ | 7/16 [00:01<00:02, 3.85it/s]Process SpawnProcess-4:
Traceback (most recent call last):
File "/home/siw669/anaconda3/envs/nnunet/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/siw669/anaconda3/envs/nnunet/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/siw669/nnunet/nnunetv2/inference/data_iterators.py", line 57, in preprocess_fromfiles_save_to_queue
raise e
File "/home/siw669/nnunet/nnunetv2/inference/data_iterators.py", line 31, in preprocess_fromfiles_save_to_queue
data, seg, data_properties = preprocessor.run_case(list_of_lists[idx],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/siw669/nnunet/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 139, in run_case
data, seg = self.run_case_npy(data, seg, data_properties, plans_manager, configuration_manager,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/siw669/nnunet/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 109, in run_case_npy
if np.max(seg) > 127:
^^^^^^^^^^^
File "/home/siw669/anaconda3/envs/nnunet/lib/python3.12/site-packages/numpy/core/fromnumeric.py", line 2810, in max
return _wrapreduction(a, np.maximum, 'max', axis, None, out,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/siw669/anaconda3/envs/nnunet/lib/python3.12/site-packages/numpy/core/fromnumeric.py", line 88, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: zero-size array to reduction operation maximum which has no identity
100%|███████████████████████████████████████████| 16/16 [00:04<00:00, 3.89it/s]
100%|███████████████████████████████████████████| 16/16 [00:04<00:00, 3.69it/s]
100%|███████████████████████████████████████████| 16/16 [00:04<00:00, 3.88it/s]
100%|███████████████████████████████████████████| 16/16 [00:04<00:00, 3.88it/s]
sending off prediction to background worker for resampling and export
done with case_00676
Traceback (most recent call last):
File "/home/siw669/anaconda3/envs/nnunet/lib/python3.12/multiprocessing/resource_sharer.py", line 138, in _serve
Traceback (most recent call last):
File "/home/siw669/anaconda3/envs/nnunet/bin/nnUNetv2_predict", line 8, in
Hey @lolawang22
This seems very odd at first and should not happen on KiTS2023. Could you visually inspect case_00676 and see if everything seems normal, as well as the data type etc? Does it work on the other cases?
Hi @mrokuss
Thank you for replying. I trained the model with KiTS2023 data but used it to predict images from other data sources. I did visually inspect that case_00676 might have too few slices and the rotation might not be correct. This happened from time to time and I wonder if there's a way to skip abnormal images when doing prediction? I also wonder if there are any filtering criteria for the images I could add before doing prediction to avoid this kind of error. Thanks!
Hey @lolawang22
Generally this should not happen that's why there is no such filtering by default. If you want so set up a filtering (e.g. by image size or spacing) then you would have to write your own sanity check script beforehand. Of course you could also include a try-except block inside the predictor but i regard this as a rather hacky solution. Hope this still helps!