Bad results!

[ShuyUSTC]

I run the inference code with the default arguments using the following video, but it reconstruct a band mesh with ugly face and misaligned hand pose! I'm wondering whether the result is correct?

https://github.com/ttxskk/AiOS/assets/124037240/cde524d3-b826-4508-9dd3-27fa16f1756e

https://github.com/ttxskk/AiOS/assets/124037240/213f7b0f-ab18-44ba-93b4-9821ddc5c3b8

[ttxskk]

Hi @ShuyUSTC , Thank you for your feedback.

1. For the issue with the collapsed face, we also noticed this problem recently, and we have identified that it was due to not filtering out related supervisions for the face marked as invalid in Ubody (please see the attached fig). So, if your test video is similar to this data, you may get a collapsed face. We have fixed this problem, and we will release a new checkpoint soon.

2. As for the misaligned hand pose, this issue might also be similar to the above one in that we didn't use valid info (as shown in the figure, where the hand is obscured and the bounding box is incorrect, yet we used this data to train our model). I will check it further and update you on any progress.

I hope this explanation can resolve your concerns. Thank you once again for your feedback, which is invaluable in helping us improve our model.

---

Train image with GT box and GT kps:

GT SMPLX overlay:

NOTE: In the provided image, both samples have face_valid=False, which indicates that the facial data is invalid. Therefore, the correct approach would be to disable the face kps, face bbox, and expression losses for both samples. Additionally, the right sample has lhand_valid=False and rhand_valid=False, indicating that both hands are invalid. For this sample, we should also disable the hand kps, hand bbox, and hand pose losses.

[zelingao98]

Same issue here. Look forward to new ckpt

[ttxskk]

Hi @jameskuma, @ShuyUSTC, I have fixed this issue in PR #15, which will be merged into the main branch soon.

[zelingao98]

@ShuyUSTC Hi, did you run the updated code? However, Issue still exists on my own data. So strange,,,

[ttxskk]

@ShuyUSTC Hi, did you run the updated code? However, Issue still exists on my own data. So strange,,,

Are you using this new checkpoint? If not, please try using it.

[zelingao98]

@ShuyUSTC Hi, did you run the updated code? However, Issue still exists on my own data. So strange,,,

Are you using this new checkpoint? If not, please try using it.

Yes, Everything works fine!! Thank you for sharing this awesome work and patient reply!

[ttxskk]

Closed.

If you have any other questions, please reopen it.