Accessing `FaceGeometry` and `VertexBufferList` from FaceLandmarkerResult in MediaPipe Task Vision

[Mehrdadjdev]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Android

MediaPipe Tasks SDK version

Vision 0.10.14

Task name (e.g. Image classification, Gesture recognition etc.)

FaceLandmarker

Programming Language and version (e.g. C++, Python, Java)

Kotlin

Describe the actual behavior

I am currently using the MediaPipe Task Vision's FaceLandmarker to detect faces in my Android application. I can successfully get the FaceLandmarkerResult, but I am unable to access FaceGeometry from it. My goal is to extract the 3D mesh data, specifically the VertexBufferList, for further processing.

Describe the expected behaviour

I expected to retrieve the FaceGeometry from FaceLandmarkerResult and then access the mesh data through getMesh() and getVertexBufferList(). This should allow me to work with the 3D landmarks in a structured manner, which is essential for my application's functionality.

Standalone code/steps you may have used to try to get what you need

Below is the code snippet I used:

    private fun loadModels() {
        val canonical = ModelRenderable
            .builder()
            .setSource(this, Uri.parse("models/canonical.glb"))
            .setIsFilamentGltf(true)
            .setAsyncLoadEnabled(true)
            .build()

             try {
                canonicalNode = Node()
                canonicalNode.setRenderable(canonical.get())
                canonicalNode.localScale = Vector3(10f, 10f, 10f)
                val rotation = Vector3(1f, 1f, 1f)
                canonicalNode.localRotation = Quaternion.axisAngle(rotation, 0f)

                canonicalNode.localPosition = Vector3(0f, 0f, -4f)
                transparentSceneView.scene.addChild(canonicalNode)
             } catch (ignore: InterruptedException) {

             } catch (ignore: ExecutionException) {

             }
        
             null
        }
    }
    private fun updateThePosition(
        faceLandmarkerResult: FaceLandmarkerResult,
    ) {
        val faceLandmarks = faceLandmarkerResult.faceLandmarks()[0]
        // i can get the 468 face landmarks.

        //val index = 6 //landmark index 6 for on the nose
        
        //val faceGeometry = faceLandmarkerResult.faceGeometry
        //val mesh = faceGeometry.getMesh()
        //val vertex = mesh.getVertexBufferList()

        canonicalNode.localPosition = Vector3(vertex[index * 5] , vertex[index * 5 + 1], vertex[index * 5 + 2])
}

However, the FaceGeometry and methods like getMesh() or getVertexBufferList() do not seem to be accessible from the FaceLandmarkerResult or its nested classes.

### Other info / Complete Logs

_No response_

[kuaashish]

Hi @kostyaby,

Do you have any guidance on retrieving the FaceGeometry and VertexBufferList from our FaceLandmarkerResult?

Thank you!!

[Mehrdadjdev]

Hi @kostyaby and @kuaashish,

Just following up on my question about accessing FaceGeometry and VertexBufferList from FaceLandmarkerResult. Any guidance or insights would be greatly appreciated!

Thank you!