mediapipe

mediapipe copied to clipboard

Please make sure that this is a bug and also refer to the troubleshooting, FAQ documentation before raising any issues.

System information (Please provide as much relevant information as possible)

• Have I written custom code (as opposed to using a stock example script provided in MediaPipe): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Linux archlinux 5.18.16-arch1-1
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: -
• Browser and version (e.g. Google Chrome, Safari) if the issue happens on browser: -
• Programming Language and version ( e.g. C++, Python, Java): C++
• MediaPipe version: v0.8.10.2
• Bazel version (if compiling from source): 5.2.0
• Solution ( e.g. FaceMesh, Pose, Holistic ): Pose
• Android Studio, NDK, SDK versions (if issue is related to building in Android environment): -
• Xcode & Tulsi version (if issue is related to building for iOS): -

Describe the current behavior: When trying to get output from pose_landmarks stream using OutputStreamPoller#Next, sometimes the output packet is empty while there's a corresponding non-empty output packet in pose_world_landmarks.

Describe the expected behavior: When the output packet of pose_world_landmarks is not empty, the output packet of pose_landmarks is also not empty.

Standalone code to reproduce the issue: Diff: https://github.com/homuler/mediapipe/commit/feabd089138188ff4604a925f56f70991fef5736

Essentially, I modified demo_run_graph_main_gpu.cc to get outputs from pose_landmarks and pose_world_landmarks synchronously.

  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller pose_landmarks_poller,
                   graph.AddOutputStreamPoller("pose_landmarks", true));
  ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller pose_world_landmarks_poller,
                   graph.AddOutputStreamPoller("pose_world_landmarks", true));

  // After starting CalculatorGraph
  while (graph_frames) {
    // ...

    mediapipe::Packet pose_landmarks_packet;
    mediapipe::Packet pose_world_landmarks_packet;
    if (!pose_landmarks_poller.Next(&pose_landmarks_packet) || !pose_world_landmarks_poller.Next(&pose_world_landmarks_packet)) {
      break;
    }
    if (pose_landmarks_packet.IsEmpty() && !pose_world_landmarks_packet.IsEmpty()) {
      LOG(WARNING) << "pose_landmarks is empty while pose_world_landmarks exists";
      LOG(WARNING) << "pose_landmarks queue: " << pose_landmarks_poller.QueueSize() << ", pose_world_landmarks queue: " << pose_world_landmarks_poller.QueueSize();
    }
    // ...
  }

Command:

bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 //mediapipe/examples/desktop/pose_tracking:pose_tracking_gpu
GLOG_logtostderr=1 ./bazel-bin/mediapipe/examples/desktop/pose_tracking/pose_tracking_gpu --calculator_graph_config_file=mediapipe/graphs/pose_tracking/pose_tracking_gpu.pbtxt

Other info / Complete Logs : I'm not sure if it's a bug, but at least I find it a bit odd. It seems that it occurs when roi and filtered_visibility are calculated almost at the same time.

I guess the following is occurring.

Calculation Steps

NOTE:

• Step B and Step C are concurrent.
• Step B is dependent on Step A.
• Step C is dependent on Step A.
• Step D is dependent on Step B and Step C.
• SwitchDemuxCalculator will be triggered whenever its input is updated, but its container node won't be necessarily triggered (it is usually triggered only when all the inputs are given).

Step A

1. An input image is added to input_video at X.
2. poselandmarkgpu__ImagePropertiesCalculator adds its output to image_size.
3. The new image_size packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator.
4. PoseLandmarkByRoiGpu adds its output to unfiltered_pose_landmarks and unfiltered_auxiliary_landmarks.
   • In practice, outputs are not added to unfiltered_pose_landmarks and unfiltered_auxiliary_landmarks simultaneously, but the order does not matter here.

Step B

1. The new unfiltered_pose_landmarks packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__SwitchDemuxCalculator
2. poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__SwitchDemuxCalculator triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__VisibilitySmoothingCalculator_2.
3. poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__VisibilitySmoothingCalculator_2 adds its output to filtered_visibility.
4. The new filtered_visibility packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator.
5. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator relays filtered_visibility to poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2.

Step C

1. The new unfiltered_auxiliary_landmarks packet triggers poselandmarkgpu__poselandmarkfiltering__LandmarksToDetectionCalculator
2. poselandmarkgpu__poselandmarkfiltering__LandmarksToDetectionCalculator adds its output to aux_detection.
3. The new aux_detection packet triggers poselandmarkgpu__poselandmarkfiltering__AlignmentPointsRectsCalculator.
4. poselandmarkgpu__poselandmarkfiltering__AlignmentPointsRectsCalculator adds its output to roi.
5. The new roi packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator.
6. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator relays roi to poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2.

Step D

1. After finishing Step B and Step C, poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2 is triggered.
2. The output of poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2 triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchMuxCalculator.
3. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchMuxCalculator relays its input to filtered_landmarks (== pose_landmarks).

The point is that Step B and Step C is concurrent. As long as filtered_visibility is calculated and relayed to LandmarksSmoothingCalculator before roi is calculated (or vice versa), this issue does not occur. However, if filtered_visibility is calculated and roi is calculated before filtered_visibility is relayed to LandmarksSmoothingCalculator, the following occurs.

Step B and C

1. poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__VisibilitySmoothingCalculator_2 adds its output to filtered_visibility (Step B # 3).
2. InputStreamHandler pops the new filtered_visibility packet from the queue and next_timestamp_bound_ for filtered_visibility is updated.
3. The new filtered_visibility packet is sent to the corresponding input streams.
4. poselandmarkgpu__poselandmarkfiltering__AlignmentPointsRectsCalculator adds its output to roi (Step C # 4).
5. InputStreamHandler pops the new roi packet from the queue and next_timestamp_bound_ for roi is updated.
6. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator is triggered by the new filtered_visibility packet.
7. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator relays each input packet to poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2.
   • image_size is left as it is since it's empty.
   • filtered_visibility is relayed to the corresponding output stream.
   • roi is empty but the next_timestamp_bound_ was updated (at the 5th step), so the timestamp bound of the corresponding output stream is also updated.
8. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2 is triggered while the roi is empty.
9. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2 does not output, but updates its output timestamp bound.
10. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchMuxCalculator updates the timestamp bound of filtered_landmarks.
11. If a client is polling the output (using OutputStreamPoller#Next), this change of the timestamp bound is detected and an empty packet is returned to the client.

If the 5th step occurs after the 7th step finishes, the timestamp bound of roi won't be updated in the 7th step and poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2 won't be triggered in the 8th step and this issue does not occur.

This issue also does not occur for world_landmarks (filtered_world_landmarks) because the corresponding SwitchContainer has only one input stream and its container node will be triggered at most once.

Workaround

After all, this issue occurs because SwitchDemuxCalculator triggers LandmarksSmoothingCalculator before all the dependent inputs are given. I think we can avoid it by using DefaultInputStreamHandler for the SwitchDemuxCalculator instead of ImmediateInputStreamHandler. When the following patch is applied (though I/F should be improved), this issue seems to be resolved.

diff --git a/mediapipe/framework/tool/switch_container.proto b/mediapipe/framework/tool/switch_container.proto
index a9c2d90..f38de33 100644
--- a/mediapipe/framework/tool/switch_container.proto
+++ b/mediapipe/framework/tool/switch_container.proto
@@ -27,4 +27,7 @@ message SwitchContainerOptions {
 
   // Use DefaultInputStreamHandler for muxing & demuxing.
   optional bool synchronize_io = 5;
+
+  optional bool synchronize_demux_io = 6;
+  optional bool synchronize_mux_io = 7;
 }
diff --git a/mediapipe/framework/tool/switch_demux_calculator.cc b/mediapipe/framework/tool/switch_demux_calculator.cc
index c4352c8..68b56bc 100644
--- a/mediapipe/framework/tool/switch_demux_calculator.cc
+++ b/mediapipe/framework/tool/switch_demux_calculator.cc
@@ -115,7 +115,7 @@ absl::Status SwitchDemuxCalculator::GetContract(CalculatorContract* cc) {
     }
   }
   auto& options = cc->Options<mediapipe::SwitchContainerOptions>();
-  if (!options.synchronize_io()) {
+  if (!options.synchronize_io() && !options.synchronize_demux_io()) {
     cc->SetInputStreamHandler("ImmediateInputStreamHandler");
   }
   cc->SetProcessTimestampBounds(true);
diff --git a/mediapipe/framework/tool/switch_mux_calculator.cc b/mediapipe/framework/tool/switch_mux_calculator.cc
index 9982ae4..a44bc00 100644
--- a/mediapipe/framework/tool/switch_mux_calculator.cc
+++ b/mediapipe/framework/tool/switch_mux_calculator.cc
@@ -154,7 +154,7 @@ absl::Status SwitchMuxCalculator::Process(CalculatorContext* cc) {
   // Update the input channel index if specified.
   channel_index_ = tool::GetChannelIndex(*cc, channel_index_);
 
-  if (options_.synchronize_io()) {
+  if (options_.synchronize_io() || options_.synchronize_mux_io()) {
     // Start with adding input signals into channel_history_ and packet_history_
     if (cc->Inputs().HasTag("ENABLE") &&
         !cc->Inputs().Tag("ENABLE").IsEmpty()) {
diff --git a/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt b/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt
index bb3665f..27d141c 100644
--- a/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt
+++ b/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt
@@ -100,6 +100,7 @@ node {
   options: {
     [mediapipe.SwitchContainerOptions.ext] {
       enable: true
+      synchronize_demux_io: true
       contained_node: {
         calculator: "LandmarksSmoothingCalculator"
         options: {

Hi @homuler , Possibly the model produce "empty packet" instead of vector of zero detections. (please try ObserveTimestamps option, in FrameProcessor. option is only available for AddMultiStreamCallback. user would need to alter code locally to use AddMultiStreamCallback, and select the ObserveTimetampBounds parameter. https://github.com/google/mediapipe/blob/master/mediapipe/java/com/google/mediapipe/components/FrameProcessor.java#L213)

@sureshdagooglecom Sorry, what am I expected to confirm? Could you please share this issue with those who are familiar with Pose or SwitchContainer?

Possibly the model produce "empty packet" instead of vector of zero detections.

The essence of this issue is that the above statement is not true. Whatever "the model" is, it does not produce this empty packet since the packet isn't in a stream but generated here for convenience.

OutputStreamPoller#Next can return an empty packet when the output is empty, and it's OK. The problem is that OutputStreamPoller#Next can return an empty packet too early when the output has not yet been calculated (or LandmarksSmoothingCalculator is executed before all the inputs are given).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

Closing as stale. Please reopen if you'd like to work on this further.

Are you satisfied with the resolution of your issue? Yes No

@sureshdagooglecom Why is the status stat:awaiting response?

Hi @ivan-grishchenko, Could you please look into this issue? Thank you!

Hello @homuler, We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions. Thank you

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

Thank you. It seems like the situation has changed, so I will close this issue. If there is a similar problem with a new solution, I will create a new issue again.

Are you satisfied with the resolution of your issue? Yes No