Annotations disappear when exporting SAM generated labels

[coelho-k]

I setup the example to use SAM model to label my segmentation dataset faster. I uploaded a few images from my local machine and was able to prompt SAM with keypoints to generate masks for the objects in it.

In the task source, I do see the annotations as present, however, once I export them to COCO format, the annotations field is empty. I even try exporting to YOLO format, and the annotations are missing there as well. Is there something I am doing wrong? I followed the guide step-by-step.

Furthermore, in a selected region's metadata for a particular mask, how can this be included in the COCO export?

Greatly appreciate the help.

[shondle]

Hi, this is addressed in this pull request. COCO and YOLO are not supported formats, as these don't support BrushLabels in Label Studio, which is what SAM is generating. Check this guide for more details.

[mfl22]

It would be nice to add this option to label-studio. Similar question was posted here: https://github.com/facebookresearch/segment-anything/issues/215

@shondle how demanding would it be to add this feature?

[shondle]

It shouldn't be too bad, we would just need to convert either the raw segmentation prediction mask PolygonLabel or convert the rle to a PolygonLabel before sending it to the Label Studio front end. Considering that I found the following, the latter may be easier.

https://github.com/ultralytics/ultralytics/blob/4aa7969e15c7d4b6062267447728c47d4858684d/ultralytics/data/converter.py#L118

[mfl22]

I just added:

        # Get polygonal segmentation
        from mask_utils import mask2poly
        poly = mask2poly(mask)

        for p_ in poly:
            label_id = ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.ascii_lowercase + string.digits))
            # Prepare segmentation - percentage format; see:
            #   https://labelstud.io/tags/polygonlabels#Sample-Results-JSON
            p_label = np.reshape(p_, (-1, 2)).tolist()
            p_label = [[cw / width * 100, ch / height * 100] for [cw, ch] in p_label]
            result_ = {
                "from_name": "labels",  # self.from_name,
                "to_name": self.to_name,
                "original_width": width,
                "original_height": height,
                "image_rotation": 0,
                "value": {
                    "points": p_label,
                    "polygonlabels": [label],
                },
                "type": "polygonlabels",
                "id": label_id,
                "readonly": False,
            }
            results.append(result_)

to https://github.com/HumanSignal/label-studio-ml-backend/blob/master/label_studio_ml/examples/segment_anything_model/segment_anything_model.py (mask2poly just converts binary mask to polygonal representation, similar as the utility from ultralytics)

It seems to be working fine.