`inference_segmentor` Fails on list of NDarrays

[alexgoft]

I'm trying to run the segmentor on a list of images (NDArrays) via the function inference_segmentor. It seems, at according to the function description, that it's possible:

def inference_segmentor(model, img):
    """Inference image(s) with the segmentor.

    Args:
        model (nn.Module): The loaded segmentor.
        imgs (str/ndarray or list[str/ndarray]): Either image files or loaded
            images.

    Returns:
        (list[Tensor]): The segmentation result.
    """
    cfg = model.cfg
    device = next(model.parameters()).device  # model device
    # build the data pipeline
    test_pipeline = [LoadImage()] + cfg.data.test.pipeline[1:]
    
    test_pipeline = Compose(test_pipeline)
    # prepare data
    data = dict(img=img)
    data = test_pipeline(data)
    data = collate([data], samples_per_gpu=1)
    if next(model.parameters()).is_cuda:
        # scatter to specified GPU
        data = scatter(data, [device])[0]
    else:
        data['img_metas'] = [i.data[0] for i in data['img_metas']]

    # forward the model
    with torch.no_grad():
        result = model(return_loss=False, rescale=True, **data)
    return result

However, i get the following error:

  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/apis/inference.py", line 89, in inference_segmentor
    data = test_pipeline(data)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/compose.py", line 41, in __call__
    data = t(data)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/apis/inference.py", line 63, in __call__
    img = mmcv.imread(results['img'])
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmcv/image/io.py", line 206, in imread
    raise TypeError('"img" must be a numpy array or a str or '
TypeError: "img" must be a numpy array or a str or a pathlib.Path object

When removing [LoadImage()] from test_pipeline in the code above, this error replaced with:

  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/apis/inference.py", line 89, in inference_segmentor
    data = test_pipeline(data)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/compose.py", line 41, in __call__
    data = t(data)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/test_time_aug.py", line 120, in __call__
    data = self.transforms(_results)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/compose.py", line 41, in __call__
    data = t(data)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/transforms.py", line 285, in __call__
    self._resize_img(results)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/transforms.py", line 241, in _resize_img
    img, scale_factor = mmcv.imrescale(
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmcv/image/geometric.py", line 242, in imrescale
    h, w = img.shape[:2]
AttributeError: 'list' object has no attribute 'shape'

The configuration used:

norm_cfg = dict(type='BN', requires_grad=True)
model = dict(
    type='EncoderDecoder',
    pretrained='open-mmlab://msra/hrnetv2_w48',
    backbone=dict(
        type='HRNet',
        norm_cfg=dict(type='SyncBN', requires_grad=True),
        norm_eval=False,
        extra=dict(
            stage1=dict(
                num_modules=1,
                num_branches=1,
                block='BOTTLENECK',
                num_blocks=(4, ),
                num_channels=(64, )),
            stage2=dict(
                num_modules=1,
                num_branches=2,
                block='BASIC',
                num_blocks=(4, 4),
                num_channels=(48, 96)),
            stage3=dict(
                num_modules=4,
                num_branches=3,
                block='BASIC',
                num_blocks=(4, 4, 4),
                num_channels=(48, 96, 192)),
            stage4=dict(
                num_modules=3,
                num_branches=4,
                block='BASIC',
                num_blocks=(4, 4, 4, 4),
                num_channels=(48, 96, 192, 384)))),
    decode_head=dict(
        type='FCNHead',
        in_channels=[48, 96, 192, 384],
        in_index=(0, 1, 2, 3),
        channels=720,
        input_transform='resize_concat',
        kernel_size=1,
        num_convs=1,
        concat_input=False,
        dropout_ratio=-1,
        num_classes=2,
        norm_cfg=dict(type='SyncBN', requires_grad=True),
        align_corners=False,
        loss_decode=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0)),
    train_cfg=dict(),
    test_cfg=dict(mode='whole'))

data = dict(
    samples_per_gpu=1,
    workers_per_gpu=1,

    test=dict(
        type='CityscapesDataset',
        data_root='data/cityscapes/',
        img_dir='leftImg8bit/val',
        ann_dir='gtFine/val',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='MultiScaleFlipAug',
                img_scale=(2048, 1024),
                flip=False,
                transforms=[
                    dict(type='Resize', keep_ratio=True),
                    dict(type='RandomFlip'),
                    dict(
                        type='Normalize',
                        mean=[123.675, 116.28, 103.53],
                        std=[58.395, 57.12, 57.375],
                        to_rgb=True),
                    dict(type='ImageToTensor', keys=['img']),
                    dict(type='Collect', keys=['img'])
                ])
        ])
)

What am I doing wrong? it seems that inference on a list of images is not really supported.

[MeowZheng]

According to the error TypeError: "img" must be a numpy array or a str or a pathlib.Path object

Please check the data type of input image

[alexgoft]

Thank you for the quick reply!

I passed a list of loaded images as specified in the documentation and I get the error above. If I pass a NumPy array of crops (of slightly different shapes) I get the a different error:

  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/apis/inference.py", line 88, in inference_segmentor
    data = test_pipeline(data)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/compose.py", line 41, in __call__
    data = t(data)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/test_time_aug.py", line 120, in __call__
    data = self.transforms(_results)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/compose.py", line 41, in __call__
    data = t(data)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/transforms.py", line 285, in __call__
    self._resize_img(results)
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmseg/datasets/pipelines/transforms.py", line 241, in _resize_img
    img, scale_factor = mmcv.imrescale(
  File "/hdd/inspection_ai/venv/lib/python3.9/site-packages/mmcv/image/geometric.py", line 242, in imrescale
    h, w = img.shape[:2]
ValueError: not enough values to unpack (expected 2, got 1)

How can I run inference_segmentor properly on a batch of loaded crops? (essentially NumPy arrays). Passing a list/NumPy array of loaded images doesn't work.

[MeowZheng]

Could you please share with us which part of the document you referred?

[janyembo]

Hi, not sure if @alexgoft is referring to this https://github.com/open-mmlab/mmsegmentation/blob/9cf2d23aa0fcb1dcf71c48fdc5943f002765e618/mmseg/apis/inference.py#L75

But I am facing the same issue of not being able to pass a List[ndarray] to inference_segmentor. Thanks!

[jinwonkim93]

Hi, I also encounter this issue. this is what i did for passing list. https://github.com/open-mmlab/mmsegmentation/pull/1849

[xiexinch]

Hi! Thank you for your quick response. I'm more curious about the reasoning behind a lack of class weighting though. Especially with the heavy imbalance of certain classes such as poles and trains in the Cityscapes dataset, it seems that class weighting would be necessary for the model to perform well as various CNN semantic segmentation architectures such as UNet or LinkNet utilize it in their experiments.