MONAI
MONAI copied to clipboard
Project-MONAI
An unexpected error happened during the inference of the pretrained segmentation model
Describe the bug Hi, when I use the monai label example app with the pretrained segmentation model for multi-label, I face this problem:
2022-08-03 11:03:33.426 | INFO | monai.transforms.utils:get_largest_connected_component_mask:961 - testtest [] testtest []
[2022-08-03 11:03:33,428] [858] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/opt/conda/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
return await self.app(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/fastapi/applications.py", line 212, in __call__
await super().__call__(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
raise exc
File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/cors.py", line 92, in __call__
await self.simple_response(scope, receive, send, request_headers=headers)
File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/cors.py", line 147, in simple_response
await self.app(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
raise exc
File "/opt/conda/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 656, in __call__
await route.handle(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 259, in handle
await self.app(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 61, in app
response = await func(request)
File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 226, in app
raw_response = await run_endpoint_function(
File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 159, in run_endpoint_function
return await dependant.call(**values)
File "/code/monailabel/endpoints/infer.py", line 177, in api_run_inference
return run_inference(background_tasks, model, image, session_id, params, file, label, output)
File "/code/monailabel/endpoints/infer.py", line 160, in run_inference
result = instance.infer(request)
File "/code/monailabel/interfaces/app.py", line 260, in infer
result_file_name, result_json = task(request)
File "/code/monailabel/interfaces/tasks/infer.py", line 299, in __call__
data = self.run_post_transforms(data, self.post_transforms(data))
File "/code/monailabel/interfaces/tasks/infer.py", line 360, in run_post_transforms
return run_transforms(data, transforms, log_prefix="POST")
File "/code/monailabel/interfaces/utils/transform.py", line 93, in run_transforms
data = t(data)
File "/opt/conda/lib/python3.8/site-packages/monai/transforms/post/dictionary.py", line 258, in __call__
d[key] = self.converter(d[key])
File "/opt/conda/lib/python3.8/site-packages/monai/transforms/post/array.py", line 351, in __call__
mask = get_largest_connected_component_mask(foreground, self.connectivity)
File "/opt/conda/lib/python3.8/site-packages/monai/transforms/utils.py", line 962, in get_largest_connected_component_mask
comp = x_label == vals[cp.ndarray.argmax(counts)]
File "cupy/_core/core.pyx", line 863, in cupy._core.core.ndarray.argmax
File "cupy/_core/core.pyx", line 880, in cupy._core.core.ndarray.argmax
File "cupy/_core/_routines_statistics.pyx", line 105, in cupy._core._routines_statistics._ndarray_argmax
File "cupy/_core/_reduction.pyx", line 564, in cupy._core._reduction._SimpleReductionKernel.__call__
File "cupy/_core/_reduction.pyx", line 331, in cupy._core._reduction._AbstractReductionKernel._call
ValueError: zero-size array to reduction operation cupy_argmax which has no identity
It seens the counts is a null list [ ] and for only one dicom image, the error happened. But i dont know why this could happen and the image looks like fine,. Can anyone help?
Hi @SachidanandAlle ,
Have you seen a similar issue in the MONAI Label app?
Thanks in advance.
Can you try other sample images.. if possible some .nii.gz ones..
Before that please share more logs/lines above.. we print image size, type for every transforms etc..
lets try to understand if its coming due what.. Looks like inference didn't result a single positive value.. that can be checked through some kind of unit test
Can you try other sample images.. if possible some .nii.gz ones..
Before that please share more logs/lines above.. we print image size, type for every transforms etc..
lets try to understand if its coming due what.. Looks like inference didn't result a single positive value.. that can be checked through some kind of unit test
Thanks for your reply, and this is the logs with more details. And in fact, this error just occured for one dcm image, and others are fine, and for now I cant get more sample images, so I am sorry for that I cant give you more infomation.
[2022-08-04 06:42:55,129] [95] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:274) - Infer Request (final): {'largest_cc': True, 'model': 'segmentation', 'image': '/tmp/tmpz71cfzjs.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint16', 'result_compress': False, 'description': 'A pre-trained model for volumetric (3D) segmentation over 3D Images', 'device': 'cuda'}
[2022-08-04 06:42:55,137] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - PRE - Run Transform(s)
[2022-08-04 06:42:55,137] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - PRE - Input Keys: ['largest_cc', 'model', 'image', 'result_extension', 'result_dtype', 'result_compress', 'description', 'device', 'image_path']
[2022-08-04 06:42:55,789] [95] [Thread-2] [INFO] (monailabel.datastore.local:538) - Invalidate count: 0
[2022-08-04 06:42:55,820] [95] [Thread-2] [INFO] (monailabel.datastore.local:538) - Invalidate count: 0
[2022-08-04 06:42:57,637] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - PRE - Transform (LoadImaged): Time: 2.4988; image: (512, 512, 119)(float32)
[2022-08-04 06:42:57,638] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - PRE - Transform (AddChanneld): Time: 0.0000; image: (1, 512, 512, 119)(float32)
[2022-08-04 06:43:01,064] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - PRE - Transform (Spacingd): Time: 3.4255; image: (1, 360, 360, 296)(float32)
[2022-08-04 06:43:01,345] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - PRE - Transform (ScaleIntensityRanged): Time: 0.2805; image: (1, 360, 360, 296)(float32)
[2022-08-04 06:43:01,345] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - PRE - Transform (EnsureTyped): Time: 0.0002; image: torch.Size([1, 360, 360, 296])(torch.float32)
[2022-08-04 06:43:01,345] [95] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:454) - Inferer:: SlidingWindowInferer => {'roi_size': (160, 160, 160), 'sw_batch_size': 1, 'overlap': 0.25, 'mode': <BlendMode.CONSTANT: 'constant'>, 'sigma_scale': 0.125, 'padding_mode': <PytorchPadMode.CONSTANT: 'constant'>, 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'roi_weight_map': None}
[2022-08-04 06:43:01,348] [95] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:364) - Infer model path: /code/apps/radiology/model/pretrained_segmentation.pt
[2022-08-04 06:43:05,049] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - POST - Run Transform(s)
[2022-08-04 06:43:05,049] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - POST - Input Keys: ['largest_cc', 'model', 'image', 'result_extension', 'result_dtype', 'result_compress', 'description', 'device', 'image_path', 'image_meta_dict', 'image_transforms', 'pred']
[2022-08-04 06:43:05,049] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - POST - Transform (EnsureTyped): Time: 0.0001; image: torch.Size([1, 360, 360, 296])(torch.float32); pred: torch.Size([14, 360, 360, 296])(torch.float32)
[2022-08-04 06:43:05,053] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - POST - Transform (Activationsd): Time: 0.0030; image: torch.Size([1, 360, 360, 296])(torch.float32); pred: torch.Size([14, 360, 360, 296])(torch.float32)
[2022-08-04 06:43:05,120] [95] [MainThread] [INFO] (monailabel.interfaces.utils.transform:99) - POST - Transform (AsDiscreted): Time: 0.0674; image: torch.Size([1, 360, 360, 296])(torch.float32); pred: torch.Size([1, 360, 360, 296])(torch.float32)
[2022-08-04 06:43:11,116] [95] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/opt/conda/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
return await self.app(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/fastapi/applications.py", line 212, in __call__
await super().__call__(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
raise exc
File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/cors.py", line 92, in __call__
await self.simple_response(scope, receive, send, request_headers=headers)
File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/cors.py", line 147, in simple_response
await self.app(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
raise exc
File "/opt/conda/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 656, in __call__
await route.handle(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 259, in handle
await self.app(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 61, in app
response = await func(request)
File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 226, in app
raw_response = await run_endpoint_function(
File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 159, in run_endpoint_function
return await dependant.call(**values)
File "/code/monailabel/endpoints/infer.py", line 177, in api_run_inference
return run_inference(background_tasks, model, image, session_id, params, file, label, output)
File "/code/monailabel/endpoints/infer.py", line 160, in run_inference
result = instance.infer(request)
File "/code/monailabel/interfaces/app.py", line 260, in infer
result_file_name, result_json = task(request)
File "/code/monailabel/interfaces/tasks/infer.py", line 299, in __call__
data = self.run_post_transforms(data, self.post_transforms(data))
File "/code/monailabel/interfaces/tasks/infer.py", line 360, in run_post_transforms
return run_transforms(data, transforms, log_prefix="POST")
File "/code/monailabel/interfaces/utils/transform.py", line 92, in run_transforms
data = t(data)
File "/opt/conda/lib/python3.8/site-packages/monai/transforms/post/dictionary.py", line 258, in __call__
d[key] = self.converter(d[key])
File "/opt/conda/lib/python3.8/site-packages/monai/transforms/post/array.py", line 351, in __call__
mask = get_largest_connected_component_mask(foreground, self.connectivity)
File "/opt/conda/lib/python3.8/site-packages/monai/transforms/utils.py", line 959, in get_largest_connected_component_mask
comp = x_label == vals[cp.ndarray.argmax(counts)]
File "cupy/_core/core.pyx", line 941, in cupy._core.core.ndarray.argmax
File "cupy/_core/core.pyx", line 958, in cupy._core.core.ndarray.argmax
File "cupy/_core/_routines_statistics.pyx", line 105, in cupy._core._routines_statistics._ndarray_argmax
File "cupy/_core/_reduction.pyx", line 567, in cupy._core._reduction._SimpleReductionKernel.__call__
File "cupy/_core/_reduction.pyx", line 332, in cupy._core._reduction._AbstractReductionKernel._call
ValueError: zero-size array to reduction operation cupy_argmax which has no identity
In addition, if I mannually set comp = false, it seens it worked fine.
And for thus image you can run none of deepedit or segmentation or segmentation spleen. model.. True?
And for thus image you can run none of deepedit or segmentation or segmentation spleen. model.. True?
This error just happened when use the pretrained segmentation model for multi-label. Other single label models worked fine.
Will it be OK if you can share the image temporarily? This will help to debug and fix the issue quickly
Will it be OK if you can share the image temporarily? This will help to debug and fix the issue quickly
I‘m sorry I cant share the image. But I noticed that this image was upside down compared to other images, that the pelvis was on top. Hope this will help you.
Them my only guess is it has not generated any non zero results after inference.. There is config to disable largest cc.. you can enable it in 3d slicer under edit/settings/monailabel (check developer mode) and it show those config options for train/infer etc..
Them my only guess is it has not generated any non zero results after inference.. There is config to disable largest cc.. you can enable it in 3d slicer under edit/settings/monailabel (check developer mode) and it show those config options for train/infer etc..
I used the OHIF instead of 3d slicer. And my MONAIlabel version is MONAI Label 0.3.2，monai version is as below:
monai-weekly[nibabel, skimage, pillow, tensorboard, gdown, ignite, torchvision, itk, tqdm, lmdb, psutil, openslide]==0.10.dev2227
The version is bit old.. and few things are improved in 4.2 version..
I suggest to uninstall existing monai and monailabel and try latest version of monailabel from pip
The version is bit old.. and few things are improved in 4.2 version..
I suggest to uninstall existing monai and monailabel and try latest version of monailabel from pip
OK. I will try the latest version, thank you very much.
Them my only guess is it has not generated any non zero results after inference.. There is config to disable largest cc.. you can enable it in 3d slicer under edit/settings/monailabel (check developer mode) and it show those config options for train/infer etc..
Discovered the same issue (ValueError: zero-size array to reduction operation cupy_argmax which has no identity) with version 0..4.2 and 3D Slicer. Disabling largest cc as discribed by @SachidanandAlle resolved the problem and segmentation is displayed.
