Issue with Human Body Segmentation

[Origin2333]

Dear @itailang

I read your paper with great interest and noticed that your model is capable of segmenting human body parts. I used the code you provided and trained the model on a human mesh dataset. To speed up the process during "Generate data for training the decoder," I used 2% of the vertices.

After training, I tested the model on the same human dataset. However, I found that it was not able to segment specific body parts. As shown in the attached image, when I perform a single positive click, the entire human body gets segmented. When I then perform a negative click, the entire body is excluded. This behavior remains the same even after several tries.

Do you have any suggestions on how to address this issue?

Thank you very much for your time and help.

[itailang]

Dear @Origin2333,

Thanks for your query. We rely on SAM for providing supervision for our model training. You can check the training data generated by the data_generation.py script and see if SAM segments the regions you are interested in.

[Origin2333]

Thank you very much for your response. As you suggested, I examined the training data and the segmentation results from SAM. SAM seems to segment the relevant regions correctly.

Could you kindly advise what aspects I should investigate next in order to identify the source of the issue?

[itailang]

With pleasure. I suggest the following steps:

1. Inspect iSeg encoder's test results. See if the auto segmentation masks of SAM using rendered iSeg features and the original SAM features are similar.
2. Train an iSeg decoder for a single click and a single view. Say, the click and the view that you have shared.
3. Inspect other views of this click as well to see if SAM's segmentation makes sense.
4. Train an iSeg decoder with a single click and multiple views. Say, for rhe click you have shared.
5. Test the segmentation result for the training vertex.
6. Test the segmentation result for a different vertex within your region of interest.

Validate that the results make sense at each step before moving on to the next.