[FR] Add support for customizing the look-and-feel of labels on a per-field basis

[brimoor]

Many users have requested fine-grained control over the colors used to render labels in the App.

Example requests:

• Use #FF6D04 for my ground_truth field
• Use #499CEF for all objects with label="cat" in any field
• Use #6D04FF for all objects with <field>=<value> in the ground_truth field

I think the right place to start with this is just to add support for these kind of customizations via the App config.

Rough example (structure needs work):

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")
dataset.evaluate_detections("predictions", gt_field="ground_truth", eval_key="eval")

session = fo.launch_app(dataset)

# Custom field colors
session.config.colors = {
    "ground_truth": {
        "color_by": "field",
        "color": "#FF6D04",
    },
    "predictions": {
        "color_by": "field",
        "color": "#499CEF",
    }
}

# Custom field value colors
session.config.colors = {
    "ground_truth": {
        "color_by": {"attribute": "eval"},
        "colors": {
            "tp": "#FF6D04",
            "fp": "#499CEF",
            "fn": "#6D04FF",
        },
        "default": "#FFFFFF",
    }
}

Comments

• If the user neglects to specify a color for a certain field/label, having a sensible fallback (eg the default color_pool strategy) will be important
• Color is one type of customization, but other L&F customizations may be possible too, such as choosing whether segmentation fields are rendered as border-only, filled, or filled-with-different style. So, session.config.colors may not be general enough here.
• Sometimes users may want to customize individual fields by name; other times they may want to customize all fields of a certain type (eg all Detection(s) fields)

[brimoor]

For context, currently the only way to customize field colors is through configuring the color pool, which can be done both for the App and when using draw_labels(), but one doesn't have control over which color is selected for each field:

import eta.core.annotations as etaa

import fiftyone as fo
import fiftyone.utils.annotations as foua
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

#
# Option 1: tell the App to use a specific color pool
#

app_config = fo.AppConfig()
app_config.color_pool = ["#FF0000", "#00FF00", "#0000FF"]

session = fo.launch_app(dataset, config=app_config)

#
# Option 2: tell `draw_labels()` to use a specific color pool
#

# Manual colormap
colormap_config = etaa.ColormapConfig(
    {
        "type": "eta.core.annotations.ManualColormap",
        "config": {"colors": ["#FF0000", "#00FF00", "#0000FF"]}
    }
)

# Also customize look-and-feel for fun
draw_config = foua.DrawConfig(
    {
        "show_all_confidences": True,
        "per_object_label_colors": False,
        "show_object_names": False,
        "show_object_attrs": False,
    }
)

dataset.limit(10).draw_labels(
    "/tmp/quickstart/",
    label_fields=["ground_truth", "predictions"],
    config=draw_config,
    colormap_config=colormap_config,
    overwrite=True,
)

[ehofesmann]

Another look and feel customization that has been requested is to be able to specify the width of bounding box lines. This can be useful depending on the clutter of detections in samples, or if working with small objects.

[brimoor]

Related: https://github.com/voxel51/fiftyone/issues/1515

[zimonitrome]

Hello @brimoor, you mentioned in #1515 that this FR might be implemented soon. Has there been any start or preparation for the implementation or are you open to pull requests?

[twmht]

any update on this?

I have similar issue for cutomizing colors (https://github.com/voxel51/fiftyone/issues/3410)

[lanzhenw]

@twmht yes, it's released in 0.21. https://docs.voxel51.com/user_guide/app.html#color-schemes-in-the-app

[twmht]

@lanzhenw

it's great to see that. Would you help me to check if this (https://github.com/voxel51/fiftyone/issues/3410) is a bug or not? I think the number of default color pools is enough for 3 classses.

[cceyda]

@brimoor Is it possible to use this feature for fo.Segmentation as well? Because my mask's values are based on the instance id number instead of class id.

Example: My mask is a 2D array with 0 for background(transparent). And 1,2,3.... for the regions of the 1st,2nd,3rd instance respectively. Along with each image&mask pair I have a list like below mapping the instance_ids to a class_name like annotation=[{"class_name":"cat","instance_id":"1"},{"class_name":"dog","instance_id":"2"}]

If I want to color cat: orange, dog:blue I can't use dataset.mask_targets method described here Because it requires there to be a constant mapping mask_value->class whereas in my case mask_value->class changes for each example since the mask values are based on the instance_id. eg: If an image has 2 cats it would be like 1->cat,2->cat, then another image with a dog&cat would be 1->dog,2->cat.

I would rather not have to save another copy of the mask in fiftyone format & I especially want to use the mask_path attribute to pass the mask (to save disk space).

So If I save my dataset like below how can I define a custom fo.ColorScheme?

for p,annotation in (paths,annotations):
    sample = fo.Sample(filepath='images/'+p)

    sample["segmentation"] = fo.Segmentation(mask_path='masks/'+p,
                                         labels={a['instance_id']:a['class_name'] for a in annotation},
                      # maybe a way to pass a color mapping per sample?
)
    dataset.add_sample(sample)

[brimoor]

Hi @cceyda 👋

Unfortunately fo.ColorScheme does not yet include support for customizing colors for Segmentation mask targets. We're planning to add that soon though!

We don't, however, have any plans to support the type of instance-based pixel scheme that you're using with the Segmentation label type. It is intended specifically for semantic segmentation, where each pixel value represents the same semantic class across images in the dataset (hence how mask_targets are implemented, as you point out).

Rather, for instance-based tasks we have Instance segmentation format.

You can convert your Segmentation values into this format very easily!

# A segmentation in your "instance" format
mask_targets = {1: "cat", 2: "dog", ...}
segmentation = fo.Segmentation(...)

# Convert to FiftyOne's instance segmentation format
instances = segmentation.to_detections(mask_targets=mask_targets)
sample = fo.Sample(filepath=..., instances=instances)

You can effectively construct different mask_targets for each sample, and the instances field will have the correct label for each instance.

[cceyda]

@brimoor glad to hear there are plans to add customizing colors for Segmentation.

I have tried the format conversion code example you gave (which is a very useful conversion to have btw.) Interestingly for me I had to use the rgb(hex) format despite my masks having 2D(w,h) shape.

rgb2hex=lambda r:'#%02x%02x%02x' % (r, r, r)
mask_targets = {rgb2hex(a['instance_id']):a["class_name"] for a in annotations}

segmentation = fo.Segmentation(mask_path=mask_image_path)
instances = segmentation.to_detections(mask_targets=mask_targets)
sample = fo.Sample(filepath=image_path, instances=instances)
print(instances)

but I see this results in mask=array(...) being saved in the sample which is something I would like to avoid as I have a pretty big dataset and I wanna save disk/memory space. I understand it is probably this way because of the Detections label not having a mask_path attribute like Segmentation and also because the masks are saved in relation to the bbox. For the time being I'll remove the mask=array(...) from the converted Detections and just rely on the bbox color for the visuals (saving along with fo.Segmentation(mask_path=...)). And maybe in the future there will be a more efficient format for saving/loading instance segmentations 🤔