Memory Leak in the HandLandmarker object using the GPU Delegate(Python M1 MacOS)

[kyleellefsen]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

Mac OS

Mobile device if the issue happens on mobile device

No response

Browser and version if the issue happens on browser

No response

Programming Language and version

Python

MediaPipe version

0.10.15

Bazel version

No response

Solution

HandLandmarker

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

No response

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

Running out of RAM and crashing

Describe the expected behaviour

not crashing

Standalone code/steps you may have used to try to get what you need

See https://github.com/google-ai-edge/mediapipe/issues/5652#issuecomment-2379932394 below

[kyleellefsen]

When I run the hand landmarker in python (see code below) on my GPU, my memory fills up until the operating system kills the python process. The entire script below takes ~5 seconds to run on my computer. (This seems similar to https://github.com/google-ai-edge/mediapipe/issues/5626 (reported 2 weeks ago))

My computer

• macOS 14.5
• Apple M1 Pro
• Memory: 16 GB

My python setup

• running inside fresh venv
• Python 3.12.3
• mediapipe==0.10.15
• numpy==1.26.4

Minimum reproducible example (memleak.py):

import time
import numpy as np
import mediapipe

def main_loop(landmarker):
    frame = np.random.randint(0, 255, (1080, 1920, 3), dtype=np.uint8)
    mp_image = mediapipe.Image(image_format=mediapipe.ImageFormat.SRGBA, data=frame)
    landmarker.detect_async(mp_image, int(1000*time.time()))

if __name__ == "__main__":
    options = mediapipe.tasks.vision.HandLandmarkerOptions(
        base_options=mediapipe.tasks.BaseOptions(
            model_asset_path='hand_landmarker.task',
            delegate=mediapipe.tasks.BaseOptions.Delegate.GPU),
        running_mode=mediapipe.tasks.vision.RunningMode.LIVE_STREAM,
        result_callback=lambda x, y, z: None)
    landmarker = mediapipe.tasks.vision.HandLandmarker.create_from_options(options)
    for idx in range(800):
        main_loop(landmarker)

> python memleak.py
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1727466093.454681 28633460 gl_context.cc:357] GL version: 2.1 (2.1 Metal - 88.1), renderer: Apple M1 Pro
INFO: Created TensorFlow Lite delegate for Metal.
W0000 00:00:1727466093.540934 28633535 landmark_projection_calculator.cc:186] Using NORM_RECT without IMAGE_DIMENSIONS is only supported for the square ROI. Provide IMAGE_DIMENSIONS or use PROJECTION_MATRIX.
fish: Job 1, 'python memleak.py' terminated by signal SIGKILL (Forced quit)

My memory spikes when I run this, and stays high until the program closes. If I extend the loop in the code above from 800 to 10000, the program is shut down by the OS. I can run it in an interactive environment (ipython) and I can see that the memory stays high until I del landmarker, at which point all the memory is released. From this I can state with certainty that the problem is with the HandLandmarker object. Also, if I switch the delegate to Delegate.CPU, there is no problem.

[kyleellefsen]

In https://github.com/google-ai-edge/mediapipe/issues/5626, I noticed the running_mode was set to VIDEO, whereas this bug has a problem with LIVE_STREAM. I tested out both (see code below) and noticed that neither of them release memory automatically. I also tested the IMAGE running mode (not shown) and it has the same problem.

import time
import numpy as np
import mediapipe

def run_video():
    options = mediapipe.tasks.vision.HandLandmarkerOptions(
        base_options=mediapipe.tasks.BaseOptions(
            model_asset_path='hand_landmarker.task',
            delegate=mediapipe.tasks.BaseOptions.Delegate.GPU),
        running_mode=mediapipe.tasks.vision.RunningMode.VIDEO)
    landmarker = mediapipe.tasks.vision.HandLandmarker.create_from_options(options)
    for idx in range(800):
        frame = np.random.randint(0, 255, (1080, 1920, 3), dtype=np.uint8)
        mp_image = mediapipe.Image(image_format=mediapipe.ImageFormat.SRGBA, data=frame)
        landmarker.detect_for_video(mp_image, int(1000*time.time()))
    return landmarker

def run_live():
    options = mediapipe.tasks.vision.HandLandmarkerOptions(
        base_options=mediapipe.tasks.BaseOptions(
            model_asset_path='hand_landmarker.task',
            delegate=mediapipe.tasks.BaseOptions.Delegate.GPU),
        running_mode=mediapipe.tasks.vision.RunningMode.LIVE_STREAM,
        result_callback=lambda x, y, z: None)
    landmarker = mediapipe.tasks.vision.HandLandmarker.create_from_options(options)
    for idx in range(800):
        frame = np.random.randint(0, 255, (1080, 1920, 3), dtype=np.uint8)
        mp_image = mediapipe.Image(image_format=mediapipe.ImageFormat.SRGBA, data=frame)
        landmarker.detect_async(mp_image, int(1000*time.time()))
    return landmarker

if __name__ == '__main__':
    landmarker_video = run_video()
    del landmarker_video  # memory is released only after landmarker is deleted
    landmarker_live = run_live()
    del landmarker_live  # same here

[kuaashish]

Hi @kyleellefsen,

Could you please verify if the issue is still present in the most recent version 0.10.18 and let us know if status of this now?

Thank you!!

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[kyleellefsen]

.

[michaeldlutlee1]

0.10.18 have the same error，on my mac m2

[PixenKnight]

0.10.18 on M1 also has high memory usage, using livestream mode with a GPU delegate

[gangfang]

This is my workaround for now: in the main loop, I force a reinstantiation of the lanmarker every 1000 frames.

            # Process frame with new API using detect_for_video
            # Re-instantiate landmarker every 1000 frames to prevent memory leaks
            if frame_idx % 1000 == 0 and frame_idx > 0:
                self.landmarker.close()
                base_options = python.BaseOptions(
                    model_asset_path='landmarker_heavy.task', 
                    delegate=mp.tasks.BaseOptions.Delegate.GPU
                )
                options = vision.PoseLandmarkerOptions(
                    base_options=base_options,
                    output_segmentation_masks=False,
                    min_pose_detection_confidence=0.5,
                    min_pose_presence_confidence=0.5,
                    min_tracking_confidence=0.5,
                    running_mode=vision.RunningMode.VIDEO
                )
               self.landmarker = mp.tasks.vision.Landmarker.create_from_options(options)

[gangfang]

Hi kuaashish, 0.10.20 has the same error, on my mac m2.

[kitschpatrol]

I'm also seeing this issue in PoseLandmarker with the GPU delegate on an M3 Mac with Mediapipe 0.10.21.

[son-bii]

I also have the same issue when setting delegate as GPU. Device: Macmini M4 Mediapipe: 0.10.21

Has anyone solved this problem?

[VimalMollyn]

Still a bug on 0.10.21 for me as well.