Strange mesh in testing

[dldaisy]

Hi, I run testing on wild images and found sometimes it will output extremely unnatural body mesh. Here is the input image: And below are the screenshots of the reconstructed image from front and side view: As you can see, the body seemed to be twisted 180 degree near the joint SPINE_3, as the result, the upper body is inside-out. I also test the model on several other images and find that this problem would very much likely to occur when the input images don't include full body(for example, only partial body or half body). May I ask why this weird problem exists? Thanks a lot!

[Jeff-sjtu]

Hi @dldaisy,

In the case of partial observation, the current HybrIK model cannot handle it correctly. Since the joint rotations are calculated based on the detected keypoints, when the keypoint is missing from the input image, the rotation is incorrect.

For the abnormal twist, maybe there are bugs in calculating the rotation of SPINE_3. I will look into this problem.

[dldaisy]

Got it, Thank you!

[puppet101]

@dldaisy can you share your testing code for in the wild images?

[dldaisy]

hello @puppet101, My test code is adapted from https://github.com/Jeff-sjtu/HybrIK/issues/30#issuecomment-877009556

[coollx]

Hi @dldaisy ,after browsing all the issues and posts I still have no idea to generating the test code. Do you mind to share your own testing code for the wild images? This would help a lot. Thanks in advance!

[biansy000]

hello @dldaisy , we found a bug in calculating the rotation matrix for spine3 in inference, and the updated code is uploaded in fix_wrong_rotmat branch. The bug is probably the cause of the abnormal twist.

[dldaisy]

@biansy000 Thanks for the update. I think the bug is due to the matrix recovered from svd is not guaranteed to have positive determinant right? Which means if the determinant is negative, the matrix will embed a reflection operation in it instead of being a real rotation matrix. The great code piece updated definitely solve the problem by modifying the matrix to a rotation matrix. But I still have a question: after this modification, would the optimality of the rotation matrix got from svd still be guaranteed, since it put extra constraint to your proof in the supplementary material? Thanks a lot.

[biansy000]

@dldaisy Yes, the bug is due to the negative determinant of matrix, and you can refer to ref1 or ref2 for further information. But neither of the references make a detailed proof.

[dldaisy]

I see. Thanks a lot.