Error when trying to set deligate to GPU on Mac

[wolfiex]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

Mac M1 Pro

MediaPipe Tasks SDK version

10.9

Task name (e.g. Image classification, Gesture recognition etc.)

Gesture Recognition

Programming Language and version (e.g. C++, Python, Java)

Python

Describe the actual behavior

Code Crashes on Runtime when using GPU deligate

Describe the expected behaviour

Code runs successfully as with the CPU

Standalone code/steps you may have used to try to get what you need

# As per the gesture recogniser example;

options = GestureRecognizerOptions(
        base_options=BaseOptions(model_asset_path='./gesture_recognizer.task',
                                                delegate=BaseOptions.Delegate.GPU),
        running_mode=VisionRunningMode.LIVE_STREAM,
        num_hands=2,
    )

    with GestureRecognizer.create_from_options(options) as recognizer:
        ...

Other info / Complete Logs

F0000 00:00:1707597159.789874       2 gpu_buffer_storage_cv_pixel_buffer.cc:154] Check failed: status_or_buffer is OK (UNKNOWN: ; unsupported ImageFrame format: 1) 
*** Check failure stack trace: ***
    @        0x106aa8628  absl::lts_20230125::log_internal::LogMessage::SendToLog()
    @        0x106aa7fc0  absl::lts_20230125::log_internal::LogMessage::Flush()
    @        0x106aa8a50  absl::lts_20230125::log_internal::LogMessageFatal::~LogMessageFatal()
    @        0x106aa8a78  absl::lts_20230125::log_internal::LogMessageFatal::~LogMessageFatal()
    @        0x106502200  mediapipe::ConvertFromImageFrame()
    @        0x106502a6c  std::__1::__invoke_void_return_wrapper<>::__call<>()
    @        0x106501180  std::__1::invoke<>()
    @        0x1064fff34  mediapipe::GpuBuffer::StorageHolder::GetStorageForView()
    @        0x10650099c  mediapipe::GpuBuffer::GetStorageForViewOrDie()
    @        0x10648479c  mediapipe::Image::ConvertToGpu()
    @        0x10617d590  std::__1::__function::__func<>::operator()()
    @        0x10650689c  mediapipe::GlContext::SwitchContextAndRun()
    @        0x106506064  mediapipe::GlContext::Run()
    @        0x106187858  mediapipe::GlCalculatorHelper::RunInGlContext()
    @        0x106187a24  mediapipe::GlCalculatorHelper::RunInGlContext()
    @        0x10617d29c  mediapipe::GlCalculatorHelper::RunInGlContext<>()
    @        0x10617c8c4  mediapipe::api2::ImageCloneCalculator::Process()
    @        0x10651760c  mediapipe::CalculatorNode::ProcessNode()
    @        0x1064f61f4  mediapipe::internal::SchedulerQueue::RunCalculatorNode()
    @        0x1064f5cc8  mediapipe::internal::SchedulerQueue::RunNextTask()
    @        0x10650f964  mediapipe::ThreadPool::RunWorker()
    @        0x10650f370  mediapipe::ThreadPool::WorkerThread::ThreadBody()
    @        0x18f99d034  _pthread_start
    @        0x18f997e3c  thread_start
Abort trap: 6

[wolfiex]

Changing delegate to 1 does however run.

base_options=BaseOptions(model_asset_path='./gesture_recognizer.task',
                                                delegate=1),

[kuaashish]

Hi @wolfiex,

Thanks for reporting this, We are looking into issue. When you set delegate=1, it must show a similar error as the GPU, but right now, it is using the CPU. Further, With GPU since you are getting this error, We are assigning to right owner.

[twoertwein]

I got the same stack trace on a Mac M2 when using FaceLandmarker. Changing the image format to SRGBA fixed the issue for me:

frame = mp.Image(image_format=mp.ImageFormat.SRGBA, data=cv2.cvtColor(frame, cv2.COLOR_BGR2RGBA))

but it would be great if the GPU would support RGB (or BGR!) - I assume that could save a little bit of time? Seeing only an improvement from 35s (CPU) to 29s (GPU) for my test video.

[schmidt-sebastian]

Our Metal implementation only supports formats with alpha channel. We should document this better (and provide a better error). Thank you for raising this.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No