Semantic segmentation option for BrushLabel

[csaroff]

Is your feature request related to a problem? Please describe. Currently labelstudio only allows for instance-segmentation with brushlabels. It would be great if it would also support the more popular semantic segmentation brushlabel which would only allow one class to be labeled per pixel.

Describe the solution you'd like We should have a parameter within the brushlabel config for segmentationType.

The new params list would be:

Param	Type	Default	Description
name	string		Name of the element
toName	string		Name of the image to label
[choice]	single | multiple	single	Configure whether the data labeler can select one or multiple labels
[maxUsages]	number		Maximum number of times a label can be used per task
[showInline]	boolean	true	Show labels in the same visual line
[segmentationType]	instance | semantic | panoptic	instance	Configure the type of brush labeling task(semantic segmentation, instance segmentation, panoptic segmentation)

<BrushLabels name="tumors" segmentationType="semantic" toName="image">
</BrushLabels>

Selecting semantic segmentation would only allow one region per label and overlapping one region over another would overwrite the former region. This would prevent issues where two classes/labels overlap.

Describe alternatives you've considered When importing the annotations during model training, I've considered just merging multiple regions of the same class together into one class. Unfortunately, this doesn't prevent overlap of two different classes/labels and there's no obvious heuristic for resolving overlapping when trying to convert from instance segmentation labels to semantic segmentation labels.

Additional context Related to https://github.com/heartexlabs/label-studio/issues/1246

[makseq]

@csaroff Do you mean the last stroke should exclude previous? Or the first stroke is always there?

[csaroff]

@makseq I would favor the current stroke overwriting old strokes on the pixels where they overlap. Does that make sense?

[d4l3k]

@makseq is there any ETA for this? It would be a big help for data labeling for driving tasks

[makseq]

@d4l3k I know how to solve it on the export step, but right now we don't have a free hands :-( if you are familiar with python, I will help to make a PR with the semantic segmentation output on the export step.

[d4l3k]

Ah, I can always do it in post if the fix is on the export side. Just would be nice to have it in the UI to avoid ambiguity when there's overlap. Also could highlight gaps

[makseq]

We need to fix export - it should save the order of brush layers. Knowing this we can understand the latest pixel mask value.

[csaroff]

IMO dealing with the overlap at export time isn't really solving the problem. The annotators themselves need to see how the overlap is handled in the UI in real time as it will change their annotation behavior

[makseq]

They see the latest strokes over previous, don't they? however these order can be lost at export step.

[csaroff]

They will see the strokes overlapping. If this feature is implemented at the export step it isn't visually clear that the latest stroke erases the previous stroke.

Even if the annotators know the export behavior, it gets pretty difficult to tease apart what will happen at export time when there are more than two segmentations in an image.

[omaralvarez]

Same problem here, fixing this would be amazing.

[SprAJR]

Could we create a 'layer' or 'weight' parameter, which would be an 'int' for each segmentation mask ? Higher positive numbers would override and deconflict lower mask numbers in preview/export. Perhaps provide an override weight for each unique annotation?

[makseq]

I guess exported layers have the order in their filenames, maybe this order can help with overlap understanding?

[csaroff]

I guess exported layers have the order in their filenames, maybe this order can help with overlap understanding?

I don't the annotators will have context on how the data will be formatted after it's exported. I don't think they should need to either.

This should really be addressed in the UI itself. Even if there is a deterministic way of converting labelstudio's instance-segmentation labels into semantic segmentation labels post-export, it doesn't address that the annotators can't see the resultant segmentation mask while they're labeling.

[MitanshuShaBa]

@makseq

It would help either way, if the feature can be added as an export step. Overlapping layers could convey the overriding behaviour.

If it can be reflected in the UI, it would be better