FaceLandmarker is non-deterministic in VIDEO/LIVE-STREAM mode

[twoertwein]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Mac M2

Mobile device if the issue happens on mobile device

No response

Browser and version if the issue happens on browser

No response

Programming Language and version

python

MediaPipe version

0.10.9 (latest version working on Mac - https://github.com/google/mediapipe/issues/5168)

Bazel version

No response

Solution

FaceLandmarker

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

No response

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

The same image should produce the same results.

Describe the expected behaviour

Consecutive calls to detect_for_video return different results for the same image.

Standalone code/steps you may have used to try to get what you need

from mediapipe import Image
from mediapipe.tasks.python import BaseOptions
from mediapipe.tasks.python.vision import (
    FaceLandmarker,
    FaceLandmarkerOptions,
    RunningMode,
)

base_options = BaseOptions(
    model_asset_path="face_landmarker_v2_with_blendshapes.task"
)
options = FaceLandmarkerOptions(
    base_options=base_options,
    running_mode=RunningMode.VIDEO, # LIVE_STREAM has the same issue as video
    output_face_blendshapes=True,
    num_faces=1,
)

image = Image.create_from_file("neutral_face_photo.png")

with FaceLandmarker.create_from_options(options) as model:
    for timestamp in range(10):
        first_blendshape = model.detect_for_video(image, timestamp).face_blendshapes[0][0].score
        print(timestamp, first_blendshape)
        # detect / IMAGE: 3.8817524909973145e-06

Other info / Complete Logs

0 3.8817524909973145e-06
1 3.895277131960029e-06
2 3.8954626688791905e-06
3 3.896072030329378e-06
4 3.8940884223848116e-06
5 3.894496785505908e-06
6 3.8950020098127425e-06
7 3.893160283041652e-06
8 3.893947450706037e-06
9 3.89325668948004e-06

[twoertwein]

maybe related https://github.com/google/mediapipe/issues/4981

[kuaashish]

Hi @twoertwein,

Could you kindly include additional pointers, such as reference images or videos, to help us better understand the issue from our end? This will aid in gaining a clearer understanding of the challenges you are facing.

Thank you!!

[twoertwein]

I typically work with low-resolution webcam videos (640x480) but the minimal reproducible example above uses the following test image (but I expect this to be an issue with most images containing a face): https://stock.adobe.com/search?k=neutral+face&asset_id=342851651

[kuaashish]

Hi @twoertwein,

Could you please confirm whether the issue has been resolved on your end, or if you still require assistance from our end?

Thank you!!

[twoertwein]

The issue still persists with version 0.10.13