mediapipe

mediapipe copied to clipboard

System information (Please provide as much relevant information as possible)

• Have I written custom code (as opposed to using a stock example script provided in MediaPipe): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Android 12
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Realme X7 5G
• Programming Language and version ( e.g. C++, Python, Java): Kotlin
• MediaPipe version: v0.8.10
• Bazel version (if compiling from source): bazel 5.2.0
• Solution ( e.g. FaceMesh, Pose, Holistic ): Pose tracker

Describe the expected behavior: I'm running pose tracker on my device, the Android phone is pretty new one and is capable of 30fps processing. The issue is I'm not getting 30fps on my phone for pose tracker, so when I profiled I'm can see the fps is too low for each calculator (8 FPS).

Trace file - 9.zip My graph looks like this -

input_stream: "image"

# Output image with rendered results. (GpuBuffer)
# output_stream: "output_video"
# Pose landmarks. (NormalizedLandmarkList)
output_stream: "pose_landmarks"

output_stream: "throttled_input_video"

node {
  calculator: "FromImageCalculator"
  input_stream: "IMAGE:image"
  output_stream: "IMAGE_GPU:input_video"
  output_stream: "SOURCE_ON_GPU:is_gpu_image"
}

# Throttles the images flowing downstream for flow control. It passes through
# the very first incoming image unaltered, and waits for downstream nodes
# (calculators and subgraphs) in the graph to finish their tasks before it
# passes through another image. All images that come in while waiting are
# dropped, limiting the number of in-flight images in most part of the graph to
# 1. This prevents the downstream nodes from queuing up incoming images and data
# excessively, which leads to increased latency and memory usage, unwanted in
# real-time mobile applications. It also eliminates unnecessarily computation,
# e.g., the output produced by a node may get dropped downstream if the
# subsequent nodes are still busy processing previous inputs.

# https://github.com/google/mediapipe/blob/4a20e9909d55838d5630366ce719844cf06ae85c/mediapipe/calculators/core/flow_limiter_calculator.cc#L39
node {
  calculator: "FlowLimiterCalculator"
  input_stream: "input_video"
  input_stream: "FINISHED:pose_landmarks"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video"
  options: {
    [mediapipe.FlowLimiterCalculatorOptions.ext] {
      max_in_flight: 1
      max_in_queue: 1
    }
  }
}


# Subgraph that detects poses and corresponding landmarks.
node {
  calculator: "PoseLandmarkGpu"
  input_side_packet: "ENABLE_SEGMENTATION:enable_segmentation"
  input_side_packet: "MODEL_COMPLEXITY:model_complexity"
  input_side_packet: "SMOOTH_LANDMARKS:smooth_landmarks"
  input_side_packet: "USE_PREV_LANDMARKS:use_prev_landmarks"
  input_stream: "IMAGE:throttled_input_video"
  output_stream: "LANDMARKS:pose_landmarks"
  output_stream: "SEGMENTATION_MASK:segmentation_mask"
  output_stream: "DETECTION:pose_detection"
  output_stream: "ROI_FROM_LANDMARKS:roi_from_landmarks"
}

I'm using pose_detection.tflite and pose_landmark_litetflite.

Other info / Complete Logs :

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached:

Trace file - 9.zip

@kostyaby Any insights on this ?

Hey @Gojo1729,

Unfortunately I'm not involved in the pose tracker development, so it's hard to tell why is it giving you 9 FPS. I agree that Realme X7 5G is a decent Android phone (geekbench), so I'd be expecting 30 FPS from it too - but I'm not sure I fully understand the current performance profile for the pose tracker

@ivan-grishchenko any ideas?

Hey @ivan-grishchenko any suggestions ?

@ivan-grishchenko any suggestions on this ?

Hi @Gojo1729 , MediaPipe graphs will queue/drop frames when the input is faster than the system can handle. The more powerful the device (or more simple the graph), the less frames dropped/queued, the higher fps can be achieved.

Hi @sureshdagooglecom , I know that Medipipe handles the FPS based on the device it's running on, but my device is a new one and it's capable to handle 30fps easily. https://browser.geekbench.com/android_devices/realme-7-5g, I am able to run face detection in 30fps

Hi @kostyaby , is there anyone else who can provide some insights on this ?

bazel build -c opt. Add "-c opt" when build.

@luan78zaoha You mean while building AAR file ? I am already using that - bazel build -c opt --strip=ALWAYS --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a --legacy_whole_archive=0 --features=-legacy_whole_archive --copt=-fvisibility=hidden --copt=-ffunction-sections --copt=-fdata-sections --copt=-fstack-protector --copt=-Oz --copt=-fomit-frame-pointer --copt=-DABSL_MIN_LOG_LEVEL=2 --linkopt=-Wl,--gc-sections,--strip-all //mediapipe/java/com/google/mediapipe/solutioncore:solution_core.aar

and even while building the graph - bazel build -c opt mediapipe/graphs/pose_tracking:pose_tracking_gpu

@ivan-grishchenko Any updates on this ? 🙏

We are also having around 5-10fps, with a very similar graph to @Gojo1729 's, across multiple decent Android devices (samsung A11, samsung A70 etc). (Same model runs quite fine on iOS; around 60fps on XR). Is this expected?

Hello @Gojo1729, We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions. Thank you

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

This issue was closed due to lack of activity after being marked stale for past 7 days.

Are you satisfied with the resolution of your issue? Yes No