Expose runtime metric face mesh in FaceLandmarker results

[kentoh]

MediaPipe Solution (you are using)

Mediapipe Tasks Vision Face Landmarker (Web)

Programming language

Javascript/Typescript

Are you willing to contribute it

Yes

Describe the feature and the current behaviour/state

The FaceGeometry module currently returns both the face pose transformation and the runtime face mesh as part of the FaceGeometry proto object, both important primitives for typical AR applications. However, the newer Face Landmarker (Tasks Vision) only exposes the former (in the form of facialTransformationMatrices) as part of FaceLandmarkerResult.

IMO, It would be beneficial and I believe reasonably trivial to also expose and return the runtime metric mesh as well, so that it can be more accurately used as a face occluder, since the runtime mesh generally conforms better to the actual detected face contours versus using the canonical face model which is based on the average face. Also, there might be use cases where we would want to more precisely align/pin objects against the actual runtime mesh.

Will this change the current API? How?

Yes, this will return an additional payload in the FaceLandmarkerResult class containing the XYZ/UV vertex buffer of the runtime mesh.

Who will benefit with this feature?

Anyone building web AR applications with Mediapipe FaceLandmarker where (1) having an accurate occluder that conforms closely to the face is needed and/or (2) precise pinning of objects based on runtime landmark positions.

Please specify the use cases for this feature

Using the dynamic face mesh as an accurate occluder. Positioning objects more precisely against the dynamic runtime landmarks in 3D metric space.

Any Other info

I did some monkey-patching on my own here to retrieve the mesh and have verified that it works. However it would be great if there is a way we can officially support it! Thank you.