[head posture] Wrong 3d world data when face is located on side

[remmel]

My 3d world face is desfigured when I'm located in the side of the camera (top; bottom; left; right). FYI, the projected face texture is still correct (the one produced by mediapipe matches my camera projection on the mesh). Thus it's look like this is only the depth of the landmark which is incorrect (distance from the pinhole to that landmark).

Note that my focalLengh is set https://github.com/remmel/mediapipeDemos/blob/main/head_posture_static.py

Also, this is good to note that when I'm located in the center; the 3d world face is impressively good (less than ~1cm depth error! I'm using a Kinect v2 and compare them to its depth).

Gif of the correct face (center) and deformed face (left) - the left face should have the same shape than the center face :

https://sketchfab.com/3d-models/wrong-3d-world-coordinates-tech-issue-94c0d4f8c20a45089353dd9f3b6f8940 (I added the center face matching RGBD to also show you how good is the depth precision!).

Let me know what other information I could provide (create and share open3d snippet to display face; sketchab output with more faces...)

(For the record I compare https://techtee.medium.com/real-time-face-mesh-point-cloud-with-three-js-tensorflow-js-and-typescript-1f37ae844e1f which seems to output a face with good local coordinate but not https://remmel.github.io/FaceMeshFaceGeometry/examples/video/index.html - note that the 2nd webpage use a cropped image)

[remmel]

Note that when displaying the face in my webviewer using face_landmarks.landmark[idx].{x,y,z} the center face and side face have same shape (not deformed). Thus the problem seems to comes from the landmarks -> metric_landmarks transformation in get_metric_landmarks

[remmel]

FYI, this issue talks about the 3d world coordinates https://github.com/google/mediapipe/issues/3362#issuecomment-1145568221 especially https://github.com/andrechen/mediapipe/blob/ca7ddc37c77fa237576161b054c9f795d794a77f/visualstudio/face_effect/face_effect_renderer.cc#L110

[Rassibassi]

Hi,

I am not sure if I can help, this all was some time ago. When I build the facemesh algorithm in python and the landmarks -> metric_landmarks stuff, I actually copied all the cpp code from the original mediapipe into a file and tested my python version against it. It could be that the mediapipe cpp code has changes by now, though.

See this repository: https://github.com/Rassibassi/mediapipeFacegeometryPython

There is a main.cpp file and also a test_face_geometry.ipynb file which tests the cpp implementation against my python implementation.

Best, Rasmus

[remmel]

Thanks Ramus for the answser. I'll have a look. The original inspired file seems to be the (older) geometry_pipeline.cc

[Rassibassi]

Yes, you're right