Generating gt_masks and gt_labels from Semantic Segmentation Dataset

[MatCorr]

I've got a few segmentation datasets that have the structure below, with all folders being filled with .png files.

dataset/
  images/
    /train
    /val
    /test
  annotations/
    /train
    /val
    /test

However, I'm trying to train a segmentor via mmsegmentation whose train pipeline requires four keys: 'img', 'gt_semantic_seg', 'gt_masks', 'gt_labels'. The train pipeline can be seen below.

train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', reduce_zero_label=True),
    dict(type='Resize', img_scale=(2048, 512), ratio_range=(0.5, 2.0)),
    dict(type='RandomCrop', crop_size=(512, 512), cat_max_ratio=0.75),
    dict(type='RandomFlip', prob=0.5),
    dict(type='PhotoMetricDistortion'),
    dict(
        type='Normalize',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        to_rgb=True),
    dict(type='Pad', size=(512, 512), pad_val=0, seg_pad_val=255),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_semantic_seg', 'gt_masks', 'gt_labels'])
]

Sadly, from what I understand, my datasets' structure means they only have the first two keys. Because of that, when I try to fine-tune the segmentor using my dataset, I get the error below.

Original Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/mmsegmentation/mmseg/datasets/custom.py", line 215, in __getitem__
    return self.prepare_train_img(idx)
  File "/mmsegmentation/mmseg/datasets/custom.py", line 232, in prepare_train_img
    return self.pipeline(results)
  File "/mmsegmentation/mmseg/datasets/pipelines/compose.py", line 41, in __call__
    data = t(data)
  File "/mmsegmentation/mmseg/datasets/pipelines/formatting.py", line 284, in __call__
    data[key] = results[key]
KeyError: 'gt_masks'

My question is: what can I do to, based on the images and annotations I have, generate gt_masks and gt_labels for these datasets?

[IamSVP94]

@MatCorr could you share your progress with that issue, pls?

[MatCorr]

This was happening when I tried to use the Mask2Former model. I've since abandoned it in favor of an Upernet, so I didn't have to deal with this problem anymore.

However, I opened an issue about this same error in another repository and someone suggested adding dict(type='ToMask') to my training pipeline. I didn't test it, because by then I had moved on, but you can see if it works for you.

Here's the issue, by the way.

https://github.com/facebookresearch/dinov2/issues/259#issuecomment-1815262688