There is not enough memory left to execute the command

[mmaciola]

trafficstars

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

Android 12

Mobile device if the issue happens on mobile device

Lenovo M10, Unisoc T610, Mali G52

Browser and version if the issue happens on browser

No response

Programming Language and version

Java

MediaPipe version

0.10.14

Bazel version

No response

Solution

FaceDetector

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

SDK 34

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

Exception: There is not enough memory left to execute the command After a few hours I start having exceptions like below. The exception is throw for every frame for about 4 minutes and then starts running again.

Describe the expected behaviour

No exception

Standalone code/steps you may have used to try to get what you need

@Override
    public void analyze(@NonNull ImageProxy imageProxy) {
        Bitmap bitmap = null;
        try {
            bitmap = Bitmap.createBitmap(imageProxy.getWidth(), imageProxy.getHeight(), Bitmap.Config.ARGB_8888);
            bitmap.copyPixelsFromBuffer(imageProxy.getPlanes()[0].getBuffer());
            imageProxy.close();
            imageProxy = null;

            final MPImage mpImage = new BitmapImageBuilder(bitmap).build();
            final FaceDetectorResult result = mFaceDetector.detect(mpImage);
            if (!result.detections().isEmpty()) {
                processResult(bitmap, result);
            }
            mpImage.close();

        } catch (Exception e) {
            LogError(e, TAG);
        } finally {
            if (bitmap != null) bitmap.recycle();
            if (imageProxy != null) imageProxy.close();
        }
    }

Other info / Complete Logs

analyze: internal: CalculatorGraph::Run() failed: 
Calculator::Process() for node "mediapipe_tasks_vision_face_detector_facedetectorgraph__mediapipe_tasks_core_inferencesubgraph__inferencecalculator__mediapipe_tasks_vision_face_detector_facedetectorgraph__mediapipe_tasks_core_inferencesubgraph__InferenceCalculator" failed: [GL_OUT_OF_MEMORY]: There is not enough memory left to execute the command.: glMapBufferRange in external/org_tensorflow/tensorflow/lite/delegates/gpu/cl/gl_interop.cc:284

com.google.mediapipe.framework.Graph.nativeWaitUntilGraphIdle(Native Method)
com.google.mediapipe.framework.Graph.k(Unknown Source:19)
j5.e.i(Unknown Source:336)
o5.a.c(Unknown Source:54)
n.d.c(Unknown Source:6)
t.f0.run(Unknown Source:65)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
java.lang.Thread.run(Thread.java:923)

[kuaashish]

Hi @mmaciola,

Could you please provide the complete steps you are following from our documentation? This will help us understand the issue better. If necessary, we will reproduce and investigate the issue in more detail.

Thank you!!

[mmaciola]

The exception occurs after several hours of proper operation.

I am using RunningMode.IMAGE for synchronous detection.

Code is mainly based on face_detector.

Below minimal demo:

public class MVPFaceDetector implements Runnable, ImageAnalysis.Analyzer {
    private MainActivity mActivity;
    private ListenableFuture<ProcessCameraProvider> mCameraProviderFuture;
    private final ExecutorService mCameraExecutor = Executors.newSingleThreadExecutor();

    private FaceDetector mFaceDetector;
    private static final String MP_FACE_DETECTION_MODEL = "face_detection_short_range.tflite";

    private PostFaceDetection mPostFaceDetection;

    public MVPFaceDetector(final MainActivity activity) {
        mActivity = activity;

        // Setup Face Detector with Media Pipes
        if (!setupFaceDetector(mActivity)) {
            return;
        }

        // Start camera runnable
        mCameraProviderFuture = ProcessCameraProvider.getInstance(activity);
        mCameraProviderFuture.addListener(this, ContextCompat.getMainExecutor(activity));
    }

    private boolean setupFaceDetector(final Context context) {
        final BaseOptions.Builder baseOptionBuilder = BaseOptions.builder()
                .setDelegate(Delegate.GPU)
                .setModelAssetPath(MP_FACE_DETECTION_MODEL);

        try {
            final BaseOptions baseOptions = baseOptionBuilder.build();
            final FaceDetector.FaceDetectorOptions.Builder optionsBuilder =
                    FaceDetector.FaceDetectorOptions.builder()
                            .setBaseOptions(baseOptions)
                            .setMinDetectionConfidence(0.7f)
                            .setRunningMode(RunningMode.IMAGE); // Using RunningMode.IMAGE to make it synchronous

            final FaceDetector.FaceDetectorOptions options = optionsBuilder.build();
            mFaceDetector = FaceDetector.createFromOptions(context, options);
	    return true;
        } catch (Exception ignore) {}
        return false;
    }


    @Override
    public void run() {
        try {
            final ProcessCameraProvider cameraProvider = mCameraProviderFuture.get();
            final CameraSelector cameraSelector = new CameraSelector.Builder()
                    .requireLensFacing(CameraSelector.LENS_FACING_FRONT)
                    .build();
            final ImageAnalysis imageAnalysis = new ImageAnalysis.Builder()
                    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
                    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
                    .build();
            imageAnalysis.setAnalyzer(mCameraExecutor, this);

            cameraProvider.bindToLifecycle(mActivity, cameraSelector, imageAnalysis);

        } catch (Exception ignore) {}
    }

    @Override
    public void analyze(@NonNull ImageProxy imageProxy) {
        Bitmap bitmap = null;
        try {
            final long beginTime = System.currentTimeMillis();

            bitmap = Bitmap.createBitmap(imageProxy.getWidth(), imageProxy.getHeight(), Bitmap.Config.ARGB_8888);
            bitmap.copyPixelsFromBuffer(imageProxy.getPlanes()[0].getBuffer());
            imageProxy.close();
            imageProxy = null;

            final MPImage mpImage = new BitmapImageBuilder(bitmap).build();
            final FaceDetectorResult result = mFaceDetector.detect(mpImage);
            // I am using RunningMode.IMAGE, so detection is synchronous
            if (!result.detections().isEmpty()) {
                processResult(bitmap, result);
            }
            mpImage.close();

        } catch (Exception e) {
            // Exception is there
            LogError(e, TAG);
        } finally {
            if (bitmap != null) bitmap.recycle();
            if (imageProxy != null) imageProxy.close();
        }
    }
}

[Siddharth-Latthe-07]

The error you're encountering with the MediaPipe FaceDetector on your Android device is related to memory management issues, specifically OpenGL out of memory errors during prolonged usage.

Steps to Troubleshoot and Resolve: 1.Ensure Proper Resource Cleanup: Make sure that all resources are properly released after each frame. In your current code, ensure that every object, especially those related to graphics and images, is properly disposed of.

2.Reduce Memory Usage: If possible, reduce the resolution of the images being processed to lower memory usage. Consider processing frames at a lower frame rate if real-time performance is not critical.

3.Periodic Reinitialization: Periodically reinitialize the FaceDetector and other related resources to avoid long-term memory buildup.

Example Code for Improved Resource Management:-

@Override
public void analyze(@NonNull ImageProxy imageProxy) {
    Bitmap bitmap = null;
    MPImage mpImage = null;
    try {
        // Create a bitmap from the ImageProxy
        bitmap = Bitmap.createBitmap(imageProxy.getWidth(), imageProxy.getHeight(), Bitmap.Config.ARGB_8888);
        bitmap.copyPixelsFromBuffer(imageProxy.getPlanes()[0].getBuffer());

        // Close the ImageProxy to release resources
        imageProxy.close();
        imageProxy = null;

        // Build MPImage from bitmap
        mpImage = new BitmapImageBuilder(bitmap).build();

        // Detect faces
        FaceDetectorResult result = mFaceDetector.detect(mpImage);

        // Process the result if any face is detected
        if (!result.detections().isEmpty()) {
            processResult(bitmap, result);
        }

    } catch (Exception e) {
        LogError(e, TAG);
    } finally {
        // Clean up resources
        if (bitmap != null) {
            bitmap.recycle();
        }
        if (mpImage != null) {
            mpImage.close();
        }
        if (imageProxy != null) {
            imageProxy.close();
        }
    }
}

Hope This helps, Thanks

[mmaciola]

To clarify:

• my test code is exactly the same as the minimal demo above. The processResult() function does nothing - it is commented out. In such conditions, after about 6 hours, an exception occurs for each frame.
• Objects that are created by me (bitmap, mpImage and imageProxy) are closed correctly
• I use ResolutionSelector on the target code and the resolution is set to the minimum acceptable
• The memory leak must be inside the mediapipe library or inside the GPU driver (which is less likely)

Currently, for the purpose of correcting the problem, I am testing a solution that closes and reinitializes the mediapipe again after the first problem encountered, but this is only a workaround.

[Siddharth-Latthe-07]

@mmaciola You can further do,

1. Garbage Collection and Memory Management: Ensure that the garbage collector is aggressively freeing up memory. You can force garbage collection periodically, although this is generally not recommended as it can impact performance. Monitor memory usage closely to identify potential leaks. Tools like Android Profiler can help you track memory usage and identify leaks in your app.

2. Optimize MediaPipe Configuration: Check for any configuration settings in MediaPipe that might help manage memory usage better. For example, reduce the number of threads or limit the amount of memory allocated to the GPU if possible.

3. Code for Periodic Reinitialization:-

// Schedule reinitialization every 4 hours
final long REINIT_INTERVAL = 4 * 60 * 60 * 1000; // 4 hours in milliseconds

Handler handler = new Handler();
Runnable reinitRunnable = new Runnable() {
    @Override
    public void run() {
        reinitializeMediaPipe();
        handler.postDelayed(this, REINIT_INTERVAL);
    }
};

// Start the periodic reinitialization
handler.postDelayed(reinitRunnable, REINIT_INTERVAL);

private void reinitializeMediaPipe() {
    // Code to properly close and reinitialize MediaPipe
    if (mFaceDetector != null) {
        mFaceDetector.close();
    }
    mFaceDetector = FaceDetector.create(...); // Your initialization code here
}

You can also montior the logs:-

private void logMemoryUsage() {
    Runtime runtime = Runtime.getRuntime();
    long usedMemory = runtime.totalMemory() - runtime.freeMemory();
    long maxMemory = runtime.maxMemory();
    long freeMemory = runtime.freeMemory();
    
    Log.d(TAG, "Used memory: " + usedMemory / (1024 * 1024) + " MB");
    Log.d(TAG, "Max memory: " + maxMemory / (1024 * 1024) + " MB");
    Log.d(TAG, "Free memory: " + freeMemory / (1024 * 1024) + " MB");
}

// Call logMemoryUsage periodically to track memory usage
handler.postDelayed(new Runnable() {
    @Override
    public void run() {
        logMemoryUsage();
        handler.postDelayed(this, 60000); // Log memory usage every minute
    }
}, 60000);

Hope this helps Thanks

[mmaciola]

Quick finds:

• At the point where the problem starts, mFaceDetector.close(); also fails with an exception:

internal: CalculatorGraph::Run() failed: 
Calculator::Process() for node "mediapipe_tasks_vision_face_detector_facedetectorgraph__mediapipe_tasks_core_inferencesubgraph__inferencecalculator__mediapipe_tasks_vision_face_detector_facedetectorgraph__mediapipe_tasks_core_inferencesubgraph__InferenceCalculator" failed: [GL_OUT_OF_MEMORY]: There is not enough memory left to execute the command.: glMapBufferRange in external/org_tensorflow/tensorflow/lite/delegates/gpu/cl/gl_interop.cc:284
com.google.mediapipe.framework.Graph.nativeWaitUntilGraphDone(Native Method)
com.google.mediapipe.framework.Graph.j(Unknown Source:19)
com.google.mediapipe.tasks.core.a.close(Unknown Source:18)
i5.a.close(Unknown Source:2)
o5.a.c(Unknown Source:153)
n.d.c(Unknown Source:6)
t.f0.run(Unknown Source:65)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
java.lang.Thread.run(Thread.java:923)

• Memory just before crash:

Used memory: 8MB, max memory: 256MB, free memory: 2MB

• After tests, I am experiencing this issue also on other devices

[Siddharth-Latthe-07]

@mmaciola might try these:-

1. Explicit Garbage Collection: Force garbage collection periodically to ensure unused objects are cleaned up. System.gc();
2. Profile Memory Usage: Use Android Studio's memory profiler to monitor your app's memory usage over time and identify any memory leaks or spikes that occur before the crash.
3. Adjust Bitmap Handling: Ensure the bitmap handling is as efficient as possible. For example, consider using a pool of reusable bitmaps to reduce the overhead of creating and destroying them frequently. 4.Custom Resolution Handling: If possible, reduce the resolution of the images being processed to decrease the memory footprint.
4. Use MediaPipe's API Efficiently: Ensure that you are using MediaPipe's API correctly and efficiently. Check for any updates or patches that might address memory management issues.
5. Periodic Restart: As a temporary workaround, implement a periodic restart of the MediaPipe processing to clear any accumulated memory usage.

private void restartFaceDetector() {
    if (mFaceDetector != null) {
        mFaceDetector.close();
    }
    // Reinitialize the FaceDetector
    mFaceDetector = FaceDetector.create(context);
}

// Schedule a periodic restart every few hours
Handler handler = new Handler();
Runnable restartTask = new Runnable() {
    @Override
    public void run() {
        restartFaceDetector();
        handler.postDelayed(this, 4 * 60 * 60 * 1000); // 4 hours
    }
};
handler.postDelayed(restartTask, 4 * 60 * 60 * 1000); // 4 hours

7. Fallback to CPU Processing: If GPU memory issues persist, consider falling back to CPU processing for some frames to balance the load.
8. MediaPipe Issue Tracker: Report the issue on the MediaPipe GitHub issue tracker if it hasn't been reported already. Provide detailed logs and steps to reproduce to help the maintainers address the problem.

hope, this helps Thanks

[kuaashish]

Hi @mmaciola,

Could you please verify if the issue is still present in the most recent version 0.10.18 and let us know the status?

Thank you!!

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[mmaciola]

Hi @kuaashish Update from v0.10.14 to v0.10.18 fixed the memory leaking problem. Thanks!