Why not directly use instance segmentation on the 2D image?

[Lizhinwafu]

trafficstars

Why not directly use instance segmentation on the 2D image?

[meyerls]

This does only work if the instance segmentations are view consistent. It is not the case for real-world data as Grounded-SAM predicts view-inconsistent instance masks for every image. We are currently working on FruitNeRF++ to tackle this problem.

[meyerls]

With Nerfstudio do you mean the implementation of FruitNeRF with Nerfstudio or yout own implementation?

With Nerfstudio you can directly export a point cloud from your implicit representation. This can be done by casting rays from arbitrary views (so basically from all known images) and extract the point where the ray terminates or as in our case we volumetrically sample the entire space (inside a unit cube) to obtain the point cloud. Therefore you can use the ns-export function.

Does it answer your question?

[Lizhinwafu]

This does only work if the instance segmentations are view consistent. It is not the case for real-world data as Grounded-SAM predicts view-inconsistent instance masks for every image. We are currently working on FruitNeRF++ to tackle this problem.

If I have obtained consistent masks across different views, what code changes are needed?