mediapipe
mediapipe copied to clipboard
google-ai-edge
"Batch size mismatch" Error when Using Selfie Multiclass Model with GPU Delegate
Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
Yes
OS Platform and Distribution
Linux Ubuntu 20.04
MediaPipe Tasks SDK version
0.10.9
Task name (e.g. Image classification, Gesture recognition etc.)
selfie multicalss segemetation
Programming Language and version (e.g. C++, Python, Java)
Python
Describe the actual behavior
I am encountering a "Batch size mismatch" error when attempting to use the Selfie Multiclass model with GPU Delegate in MediaPipe. The error message is as follows:WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1705492784.934668 18558 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5 I0000 00:00:1705492785.005127 18620 gl_context.cc:344] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 525.147.05), renderer: NVIDIA RTX A2000 Laptop GPU/PCIe/SSE2 INFO: Created TensorFlow Lite delegate for GPU. ERROR: TfLiteGpuDelegate Prepare: Batch size mismatch, expected 1 but got 8 ERROR: Node number 175 (TfLiteGpuDelegate) failed to prepare. ERROR: Restored original execution plan after delegate application failure. E0000 00:00:1705492785.049712 18558 calculator_graph.cc:876] INTERNAL: CalculatorGraph::Run() failed: Calculator::Open() for node "mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraph__inferencecalculator__mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraph__InferenceCalculator" failed: ; RET_CHECK failure (mediapipe/calculators/tensor/inference_calculator_gl.cc:194) (interpreter_->ModifyGraphWithDelegate(delegate_.get()))==(kTfLiteOk) Traceback (most recent call last): File "mp_head.py", line 156, in
Describe the expected behaviour
I expected that when running the Selfie Multiclass model with GPU Delegate, the batch size would match the expected value, and no error would occur.
Standalone code/steps you may have used to try to get what you need
This is my code:
import cv2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
from mediapipe.framework.formats import landmark_pb2
from mediapipe import solutions
import numpy as np
import time
from typing import List
BG_COLOR = (192, 192, 192) # gray
BLACK_COLOR = (0, 0, 0) # black
MASK_COLOR = (255, 255, 255) # white
BODY_COLOR = (0, 255, 0) # green
FACE_COLOR = (255, 0, 0) # red
CLOTHES_COLOR = (255, 0, 255) # purple
BaseOptions = mp.tasks.BaseOptions
ImageSegmenter = mp.tasks.vision.ImageSegmenter
ImageSegmenterOptions = mp.tasks.vision.ImageSegmenterOptions
VisionRunningMode = mp.tasks.vision.RunningMode
class App:
def __init__(self):
self.output_image = None
self.base_options = python.BaseOptions(model_asset_path='selfie_multiclass_256x256.tflite',
delegate=python.BaseOptions.Delegate.GPU)
self.options = ImageSegmenterOptions(base_options=self.base_options,
running_mode=VisionRunningMode.LIVE_STREAM,
output_category_mask=True,
output_confidence_masks=False,
result_callback=self.mp_callback)
self.segmenter = ImageSegmenter.create_from_options(self.options)
self.latest_time_ms = 0
def update(self, frame):
t_ms = int(time.time() * 1000)
if t_ms <= self.latest_time_ms:
print("no update")
return
mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=frame)
self.segmenter.segment_async(mp_image, t_ms)
self.latest_time_ms = t_ms
def mp_callback(self, segmentation_result: List[mp.Image], rgb_image: mp.Image, timestamp_ms: int):
category_mask = segmentation_result.category_mask
image_data = rgb_image.numpy_view()
fg_image = np.zeros(image_data.shape[:2], dtype=np.uint8)
fg_image[:] = MASK_COLOR[0]
bg_image = np.zeros(image_data.shape[:2], dtype=np.uint8)
bg_image[:] = BLACK_COLOR[0]
condition1 = category_mask.numpy_view() == 1 # hair
if np.sum(condition1) == 0:
self.output_image = bg_image
else:
self.output_image = np.where(condition1, fg_image, bg_image)
if __name__ == "__main__":
app = App()
cap = cv2.VideoCapture(0)
while True:
ret, frame = cap.read()
if ret:
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
app.update(frame)
if app.output_image is not None:
cv2.imshow("Segmentation", app.output_image)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
time.sleep(0.1)
cap.release()
cv2.destroyAllWindows()
Other info / Complete Logs
No response
Hi @soonhyo ,
It looks like you are trying to modify our Task API code using the Selfie Multiclass tflite model. We haven't tested this yet, we will be unable to offer support in this case. If you are following our documentation from here, please let us know the specific guide or instructions you are using so we can better understand and address the issue.
Thank you
Thank you for your reply.
I have referred to the live stream example for Python in the MediaPipe Image Segmentation documentation. Based on the discussion in this #4984, I added delegate=python.BaseOptions.Delegate.GPU to the code. There are no significant other modifications made to the example code.
It's worth noting that the same code worked with GPU support for the hair_segmenter.tflite model. Additionally, when I converted the selfie_multiclass_256x256.tflite model to an ONNX model, it operated successfully with GPU support.
Are you able to use the model in MediaPipe Studio? https://mediapipe-studio.webapps.google.com/studio/demo/image_segmenter
You can select the same model here (multi-class segmentation) and GPU as the delegate.
Thank you for your reply, @schmidt-sebastian. Yes, it worked normally in MediaPipe Studio, but it has not worked in the Python code yet.
Same on Android 11T (21081111RG) works well with CPU delegate but catch "Batch size mismatch" error when modify to GPU. I use this device and use mediapipe studio demo, when selecting Multi-class Selfie Segmenter 256 and GPU it seems to have the same error, I think the problem is the GPU model on this device but when testing other models like Selfie Segmenter, Hair Segmenter and Deeplab V3 with the GPU delegate still working as expected
Log Details:
E Image segmenter failed to load model with error: invalid argument: CalculatorGraph::Run() failed in Run:
Calculator::Open() for node "mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraph__inferencecalculator__mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraph__InferenceCalculator" failed: Batch size mismatch, expected 1 but got 8
