MediaPipeUnityPlugin Graph: Android: 'GateCalculator' freezes, 'ImmediateMuxCalculator' internal error

Plugin Version or Commit ID

v0.10.0

Unity Version

2021.3.4f1

Your Host OS

Windows 10

Target Platform

Android

Description

I tried to create a custom graph that combines facemesh, iris, hand tracking and selfie segmentation where I can activate and deactivate any feature with gates and everything works fine on the editor but when I build on android, I get Error Unity MediaPipeException: INTERNAL: related to ImmediateMuxCalculator and if I remove the ImmediateMuxCalculator , the scene freezes on sync mode.

Code to Reproduce the issue

CPU Graph for the Editor:

input_stream: "input_video"
input_stream: "graph_selector"
input_stream: "graph_selector_hand"
input_stream: "graph_selector_selfie"
input_stream: "graph_selector_iris"

output_stream: "face_landmarks_with_iris"
output_stream: "face_rect"
output_stream: "face_detections"

# Max number of hands to detect/process. (int)
input_side_packet: "num_hands"

# Collection of detected/predicted hands, each represented as a list of
# landmarks. (std::vector<NormalizedLandmarkList>)
output_stream: "hand_landmarks"

# Collection of detected/predicted hand world landmarks.
# (std::vector<LandmarkList>)
output_stream: "hand_world_landmarks"

output_stream: "handedness"
output_stream: "palm_detections"
output_stream: "hand_rects_from_landmarks"
output_stream: "hand_rects_from_palm_detections"

output_stream: "segmentation_mask"

# Defines how many faces to detect. Iris tracking currently only handles one
# face (left and right eye), and therefore this should always be set to 1.
node {
  calculator: "ConstantSidePacketCalculator"
  output_side_packet: "PACKET:0:num_faces"
  node_options: {
    [type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
      packet { int_value: 1 }
    }
  }
}

node {
  calculator: "FlowLimiterCalculator"
  input_stream: "input_video"
  input_stream: "FINISHED:face_landmarks_with_iris"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video_face"
}

node {
  calculator: "FlowLimiterCalculator"
  input_stream: "throttled_input_video_face"
  input_stream: "FINISHED:hand_landmarks"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video_hand"
}

node {
  calculator: "FlowLimiterCalculator"
  input_stream: "throttled_input_video_hand"
  input_stream: "FINISHED:segmentation_mask"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video"
}


node: {
  calculator: "ImageTransformationCalculator"
  input_stream: "IMAGE:throttled_input_video"
  input_side_packet: "ROTATION_DEGREES:input_rotation"
  input_side_packet: "FLIP_HORIZONTALLY:input_horizontally_flipped"
  input_side_packet: "FLIP_VERTICALLY:input_vertically_flipped"
  output_stream: "IMAGE:transformed_input_video"
}

node: {
  calculator: "GateCalculator"
  input_stream: "ALLOW:graph_selector"
  input_stream: "transformed_input_video"
  output_stream: "transformed_input_video_face"
}

node: {
  calculator: "GateCalculator"
  input_stream: "ALLOW:graph_selector_hand"
  input_stream: "transformed_input_video"
  output_stream: "transformed_input_video_hand"
}

node: {
  calculator: "GateCalculator"
  input_stream: "ALLOW:graph_selector_selfie"
  input_stream: "transformed_input_video"
  output_stream: "transformed_input_video_selfie"
}

node: {
  calculator: "GateCalculator"
  input_stream: "ALLOW:graph_selector_iris"
  input_stream: "transformed_input_video"
  output_stream: "transformed_input_video_face_iris"
}

# Detects faces and corresponding landmarks.
node {
  calculator: "FaceLandmarkFrontCpu"
  input_stream: "IMAGE:transformed_input_video_face"
  input_side_packet: "NUM_FACES:num_faces"
  output_stream: "LANDMARKS:multi_face_landmarks"
  output_stream: "ROIS_FROM_LANDMARKS:face_rects_from_landmarks"
  output_stream: "DETECTIONS:face_detections"
  output_stream: "ROIS_FROM_DETECTIONS:face_rects_from_detections"
}

# Gets the very first and only face from "multi_face_landmarks" vector.
node {
  calculator: "SplitNormalizedLandmarkListVectorCalculator"
  input_stream: "multi_face_landmarks"
  output_stream: "face_landmarks"
  node_options: {
    [type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
      ranges: { begin: 0 end: 1 }
      element_only: true
    }
  }
}


# Gets the very first and only face rect from "face_rects_from_landmarks"
# vector.
node {
  calculator: "SplitNormalizedRectVectorCalculator"
  input_stream: "face_rects_from_landmarks"
  output_stream: "face_rect"
  node_options: {
    [type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
      ranges: { begin: 0 end: 1 }
      element_only: true
    }
  }
}

# Detects iris landmarks
node {
  calculator: "IrisLandmarksFromFaceLandmarksCpu"
  input_stream: "IMAGE:transformed_input_video_face_iris"
  input_stream: "FACE_LANDMARKS:face_landmarks"
  output_stream: "UPDATED_FACE_LANDMARKS:updated_face_landmarks"
  output_stream: "LEFT_EYE_IRIS_LANDMARKS:left_iris_landmarks"
  output_stream: "RIGHT_EYE_IRIS_LANDMARKS:right_iris_landmarks"
}

# Concatenate iris landmarks from both eyes.
node {
  calculator: "ConcatenateNormalizedLandmarkListCalculator"
  input_stream: "updated_face_landmarks"
  input_stream: "left_iris_landmarks"
  input_stream: "right_iris_landmarks"
  output_stream: "face_landmarks_with_iris2"
}

node: {
  calculator: "GateCalculator"
  input_stream: "ALLOW:graph_selector_iris"
  input_stream: "face_landmarks_with_iris2"
  output_stream: "face_landmarks_with_iris3"
}

node: {
  calculator: "GateCalculator"
  input_stream: "DISALLOW:graph_selector_iris"
  input_stream: "face_landmarks"
  output_stream: "face_landmarks_without_iris"
}

node {
  calculator: "ImmediateMuxCalculator"
  input_stream_handler {
    input_stream_handler: "ImmediateInputStreamHandler"
  }
  input_stream: "face_landmarks_without_iris"
  input_stream: "face_landmarks_with_iris3"
  output_stream: "face_landmarks_with_iris"
}

# Subgraph that performs hand tracking.
node {
  calculator: "HandLandmarkTrackingCpu"
  input_stream: "IMAGE:transformed_input_video_hand"
  input_side_packet: "MODEL_COMPLEXITY:model_complexity"
  input_side_packet: "NUM_HANDS:num_hands"
  output_stream: "LANDMARKS:hand_landmarks"
  output_stream: "WORLD_LANDMARKS:hand_world_landmarks"
  output_stream: "HANDEDNESS:handedness"
  output_stream: "PALM_DETECTIONS:palm_detections"
  output_stream: "HAND_ROIS_FROM_LANDMARKS:hand_rects_from_landmarks"
  output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:hand_rects_from_palm_detections"
}

# Subgraph that performs selfie segmentation.
node {
  calculator: "SelfieSegmentationCpu"
  input_stream: "IMAGE:transformed_input_video_selfie"
  output_stream: "SEGMENTATION_MASK:segmentation_mask_rotated"
}

node: {
  calculator: "ImageTransformationCalculator"
  input_stream: "IMAGE:segmentation_mask_rotated"
  input_side_packet: "ROTATION_DEGREES:output_rotation"
  input_side_packet: "FLIP_HORIZONTALLY:output_horizontally_flipped"
  input_side_packet: "FLIP_VERTICALLY:output_vertically_flipped"
  output_stream: "IMAGE:segmentation_mask"
}

OpenglES for Android:

input_stream: "input_video"
input_stream: "graph_selector"
input_stream: "graph_selector_hand"
input_stream: "graph_selector_selfie"
input_stream: "graph_selector_iris"

output_stream: "face_landmarks_with_iris"
output_stream: "face_rect"
output_stream: "face_detections"

input_side_packet: "num_hands"

output_stream: "hand_landmarks"

output_stream: "hand_world_landmarks"

output_stream: "handedness"
output_stream: "palm_detections"
output_stream: "hand_rects_from_landmarks"
output_stream: "hand_rects_from_palm_detections"

output_stream: "segmentation_mask"


node {
  calculator: "FlowLimiterCalculator"
  input_stream: "input_video"
  input_stream: "FINISHED:face_landmarks_with_iris"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video_face"
}

node {
  calculator: "FlowLimiterCalculator"
  input_stream: "throttled_input_video_face"
  input_stream: "FINISHED:hand_landmarks"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video_hand"
}

node {
  calculator: "FlowLimiterCalculator"
  input_stream: "throttled_input_video_hand"
  input_stream: "FINISHED:segmentation_mask"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video"
}

node: {
  calculator: "ImageTransformationCalculator"
  input_stream: "IMAGE_GPU:throttled_input_video"
  input_side_packet: "ROTATION_DEGREES:input_rotation"
  input_side_packet: "FLIP_HORIZONTALLY:input_horizontally_flipped"
  input_side_packet: "FLIP_VERTICALLY:input_vertically_flipped"
  output_stream: "IMAGE_GPU:transformed_input_video"
}

node {
  calculator: "ConstantSidePacketCalculator"
  output_side_packet: "PACKET:num_faces"
  node_options: {
    [type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
      packet { int_value: 1 }
    }
  }
}

node: {
  calculator: "GateCalculator"
  input_stream: "ALLOW:graph_selector"
  input_stream: "transformed_input_video"
  output_stream: "transformed_input_video_face"
}

node: {
  calculator: "GateCalculator"
  input_stream: "ALLOW:graph_selector_hand"
  input_stream: "transformed_input_video"
  output_stream: "transformed_input_video_hand"
}

node: {
  calculator: "GateCalculator"
  input_stream: "ALLOW:graph_selector_selfie"
  input_stream: "transformed_input_video"
  output_stream: "transformed_input_video_selfie"
}

node: {
  calculator: "GateCalculator"
  input_stream: "ALLOW:graph_selector_iris"
  input_stream: "transformed_input_video"
  output_stream: "transformed_input_video_face_iris"
}

node {
  calculator: "FaceLandmarkFrontGpu"
  input_stream: "IMAGE:transformed_input_video_face"
  input_side_packet: "NUM_FACES:num_faces"
  output_stream: "LANDMARKS:multi_face_landmarks"
  output_stream: "ROIS_FROM_LANDMARKS:face_rects_from_landmarks"
  output_stream: "DETECTIONS:face_detections"
  output_stream: "ROIS_FROM_DETECTIONS:face_rects_from_detections"
}

node {
  calculator: "SplitNormalizedLandmarkListVectorCalculator"
  input_stream: "multi_face_landmarks"
  output_stream: "face_landmarks"
  node_options: {
    [type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
      ranges: { begin: 0 end: 1 }
      element_only: true
    }
  }
}

node {
  calculator: "SplitNormalizedRectVectorCalculator"
  input_stream: "face_rects_from_landmarks"
  output_stream: "face_rect"
  node_options: {
    [type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
      ranges: { begin: 0 end: 1 }
      element_only: true
    }
  }
}

node {
  calculator: "IrisLandmarksFromFaceLandmarksGpu"
  input_stream: "IMAGE:transformed_input_video_face_iris"
  input_stream: "FACE_LANDMARKS:face_landmarks"
  output_stream: "UPDATED_FACE_LANDMARKS:updated_face_landmarks"
  output_stream: "LEFT_EYE_IRIS_LANDMARKS:left_iris_landmarks"
  output_stream: "RIGHT_EYE_IRIS_LANDMARKS:right_iris_landmarks"
}

node {
  calculator: "ConcatenateNormalizedLandmarkListCalculator"
  input_stream: "updated_face_landmarks"
  input_stream: "left_iris_landmarks"
  input_stream: "right_iris_landmarks"
  output_stream: "face_landmarks_with_iris2"
}

node: {
  calculator: "GateCalculator"
  input_stream: "ALLOW:graph_selector_iris"
  input_stream: "face_landmarks_with_iris2"
  output_stream: "face_landmarks_with_iris3"
}

node: {
  calculator: "GateCalculator"
  input_stream: "DISALLOW:graph_selector_iris"
  input_stream: "face_landmarks"
  output_stream: "face_landmarks_without_iris"
}

node {
  calculator: "ImmediateMuxCalculator"
  input_stream_handler {
    input_stream_handler: "ImmediateInputStreamHandler"
  }
  input_stream: "face_landmarks_without_iris"
  input_stream: "face_landmarks_with_iris3"
  output_stream: "face_landmarks_with_iris"
}

node {
  calculator: "HandLandmarkTrackingGpu"
  input_stream: "IMAGE:transformed_input_video_hand"
  input_side_packet: "MODEL_COMPLEXITY:model_complexity"
  input_side_packet: "NUM_HANDS:num_hands"
  output_stream: "LANDMARKS:hand_landmarks"
  output_stream: "WORLD_LANDMARKS:hand_world_landmarks"
  output_stream: "HANDEDNESS:handedness"
  output_stream: "PALM_DETECTIONS:palm_detections"
  output_stream: "HAND_ROIS_FROM_LANDMARKS:hand_rects_from_landmarks"
  output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:hand_rects_from_palm_detections"
}

node {
  calculator: "SelfieSegmentationGpu"
  input_stream: "IMAGE:transformed_input_video_selfie"
  output_stream: "SEGMENTATION_MASK:segmentation_mask_gpu"
}

node: {
  calculator: "ImageTransformationCalculator"
  input_stream: "IMAGE_GPU:segmentation_mask_gpu"
  input_side_packet: "ROTATION_DEGREES:output_rotation"
  input_side_packet: "FLIP_HORIZONTALLY:output_horizontally_flipped"
  input_side_packet: "FLIP_VERTICALLY:output_vertically_flipped"
  output_stream: "IMAGE_GPU:segmentation_mask_unrotated_gpu"
}

node: {
  calculator: "GpuBufferToImageFrameCalculator"
  input_stream: "segmentation_mask_unrotated_gpu"
  output_stream: "segmentation_mask"
}

Additional Context

No response

Jun 17 '22 16:06 hedi-chouaieb

And what do you want to ask?

Jun 19 '22 10:06 homuler

it works on the editor but not on android so is it a problem with my graph, with MediapipeUnityPlugin or with Mediapipe ? and if someone can help me to fix it? I will be grateful

Jun 19 '22 17:06 hedi-chouaieb

Do the graphs from the example app on this repo work on android for you first? I would start with one of those then cross-compare the graphs to see if you missed anything?

Aug 04 '22 15:08 ROBYER1

MediaPipeUnityPlugin MediaPipeUnityPlugin copied to clipboard

Graph: Android: 'GateCalculator' freezes, 'ImmediateMuxCalculator' internal error

Plugin Version or Commit ID

Unity Version

Your Host OS

Target Platform

Description

Code to Reproduce the issue

Additional Context

MediaPipeUnityPlugin
MediaPipeUnityPlugin copied to clipboard