[FR] Add support for rendering `Segmentation` primitives when the corresponding `Sample` is a pointcloud/pcd file.

[SergioRAgostinho]

Proposal Summary

Add support for rendering fiftyone.Segmentation when the corresponding Sample is a pointcloud/pcd file. Currently, semantic segmentation needs to baked into the point cloud's RGB fields. This has two drawbacks:

• If color is mapped to the RGC color fields, the user is not longer able to visualize LiDAR intensity/reflectance, since this information is rendered by sending that information to Red channel exclusively.
• If point cloud data has both semantic and instance segmentation, we need to create separate pcd files for each.

I've noticed fiftyone.Detection primitives are rendered correctly already in the 3D viewer, but the segmentation masks seem to be ignored.

Motivation

• What is the use case for this feature? It would allow quickly visualize segmentation masks (semantic, instance) for point clouds, without having to bake this information into the PCD files and without requiring unnecessary duplication to PCD files just to have multiple segmentation masks. It's just an extension of something you already provide for 2D, but to 3D.
• Why is this use case valuable to support for FiftyOne users in general? See reply above.
• Why is this use case valuable to support for your project(s) or organization? LiDAR/Radar perception is an important aspect of autonomous driving.
• Why is it currently difficult to achieve this use case? I don't know. You tell me.

What areas of FiftyOne does this feature affect?

• [x] App: FiftyOne application
• [ ] Core: Core fiftyone Python library
• [ ] Server: FiftyOne server

Details

I have a proposal in term of user experience, ideally I would like to see segmentation masks being mapped to into the "Shade by" panel of the 3D viewer. The color assignment for points should follow same policy you're using for segmentation masks in 2D

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

• [ ] Yes. I can contribute this feature independently
• [x] Yes. I would be willing to contribute this feature with guidance from the FiftyOne community
• [ ] No. I cannot contribute this feature at this time