PERF: FaceLandmarker two times slower than FaceMesh

[twoertwein]

FaceLandmarkers is two times slower than FaceMesh even though both return only the facial landmarks in their default configuration.

import mediapipe as mp
from mediapipe.python.solutions.face_mesh import FaceMesh
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
from time import time
import cv2

base_options = python.BaseOptions(
    model_asset_path="face_landmarker_v2_with_blendshapes.task",
)
options = vision.FaceLandmarkerOptions( # returns only landmarks by default
    base_options=base_options,
    running_mode=vision.RunningMode.VIDEO,
    num_faces=1,
)
face_landmarker = vision.FaceLandmarker.create_from_options(options)


face_mesh = FaceMesh(
    static_image_mode=False,
    max_num_faces=1,
    refine_landmarks=True,
)


cap = cv2.VideoCapture("video.mp4")
fps = round(cap.get(cv2.CAP_PROP_FPS))
frame_time_ms = 0

facemesh_time = 0
facelandmarker_time = 0

while True:
    ret, frame = cap.read()
    if not ret:
        break
    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

    start = time()
    face_mesh.process(frame)
    facemesh_time += time() - start

    start = time()
    face_landmarker.detect_for_video(
        mp.Image(image_format=mp.ImageFormat.SRGB, data=frame), frame_time_ms
    )
    facelandmarker_time += time() - start
    frame_time_ms += fps
print(facemesh_time, facelandmarker_time) # 13.58 vs 32.06 (on a Mac M2)

[kuaashish]

Hi @twoertwein,

This might be true because we have stopped maintaining the legacy face mesh. For now, Our aim is to keep improving the Face Landmarker task API on all platforms. Unfortunately, we will note this performance issue and inform the team to investigate.

Thank you!!

[twoertwein]

Thank you! Looking forward to a faster FaceLandmarker!

Not sure whether it is related: the speed up from switching from CPU to GPU for the FaceLandmarker is rather small (~15% faster, Mac M2) while the HandLandmarker improves by about 80%. It would be great if the GPU version of FaceLandmarker yields a larger speedup :)