Issue with SAGA Segmentation on Treehill Dataset & Aerial Images

[NiranjanaaR]

Hi everyone,

I'm working with SAGA for segmentation and followed all the steps clearly mentioned in the GitHub instructions. However, I encountered issues while running it on different datasets:

Issues Faced: Treehill Dataset:

I tried using SAGA with the Treehill dataset, but I am unable to segment anything. Clicking on points in the visualization doesn’t return any identifiable segments.

Aerial Image Dataset (Small City):

I also tested SAGA on an aerial image dataset of a small city. The quality of the generated 3D model is poor, but I expected at least some level of segmentation. However, when I click on elements in the scene, nothing is identified. Additional Concern I also noticed that clicking on objects does not highlight them at all. Is this expected behavior? Or should clicking automatically highlight/select the corresponding segment?

Request for Help I’d appreciate any insights into:

Whether the Treehill dataset is known to work with SAGA, and if there are specific settings or preprocessing steps I might be missing. Possible reasons why clicking on elements doesn’t return segmentation results or highlights. Any debugging steps I can follow to better understand what's going wrong. Since I have carefully followed all the instructions provided in the GitHub documentation, I am unsure what might be causing this issue. Any help or suggestions would be greatly appreciated!

Thanks in advance!

[Jumpat]

Hello, we haven't test SAGA on the Treehill dataset. You must ensure that 3D-GS works well on it, which is the prerequisite for running SAGA on it. Moreover, more information is required to debug.

[NiranjanaaR]

Hi [SAGA Team],

Thank you for your response. I confirm that 3D-GS works properly on the Treehill dataset (screenshot attached). However, segmentation in SAGA isn’t highlighting anything, even though:

Clicks register correctly, and the system returns clicked points. (screenshot attached) Segmentation appears to run (Segment3D button pressed! message appears). Adjusting scale and score threshold does not change the results.

Findings on Saved Segmentation (.pt File): SAGA does save a segmentation file in .pt format, but I am unsure if it contains meaningful segmentation output. Could you clarify what the .pt file should contain and how I can verify if segmentation worked correctly?

Additional Issue: Aerial Image Dataset (House Detection) I am also working on segmenting houses from aerial images of a small city, but I’m facing a similar issue:

The 3D model generated from the aerial images renders poorly, though I expected some level of segmentation.(Attached screenshot) Clicking on elements in the scene does not highlight or segment anything. Should clicking on objects always return highlighted segments if SAGA is working correctly? Questions for Debugging: Should clicking on objects always return highlighted segments? Are there additional preprocessing steps needed for Treehill and aerial images? Since SAGA saves a .pt file but segmentation is missing or incorrect, how can I verify if segmentation is working properly? I appreciate your guidance! Thank you again for your support.

Best Regards, Niranjana

[Jumpat]

Hi. You may need to check one of the boxes in Render option. The .pt file is a binary bitmap, whose entries indicate whether the corresponding 3D Gaussian belong to the target of interest.

Since SAGA saves a .pt file but segmentation is missing or incorrect, how can I verify if segmentation is working properly?

You can check its positive entries, like torch.count_nonzero, torch.all. If all entries are True, the segmentation is fail. And if not, it should work.

The preview_segmentation_in_2d option is also helpful for debugging.

[NiranjanaaR]

Hi again — thanks so much for your earlier response!

I wanted to follow up with an update and one more question.

I realized that the segmentation results were not working correctly before due to an incorrect mask being applied . Once that was fixed and the mask was properly aligned with the full point cloud, the rendering worked as expected. Thank you for pointing me in the right direction.

That said, I now have a follow-up question:

In the notebook (prompt_segmenting.ipynb), the file precomputed_mask.pt is saved quite early — before the HDBSCAN-based clustering and the language-driven 3D segmentation steps. Later, this same precomputed_mask.pt is passed into render.py to apply the segmentation.

So I’m wondering — are the clustering and language-based segmentation steps actually used to generate the final rendered mask, or are they more of an exploratory component? If the precomputed mask is saved early on, it seems like the later segmentation logic doesn’t affect the final rendering unless we manually overwrite precomputed_mask.pt.

Just wanted to clarify how those parts are intended to fit together.

Thanks again!