The output gets out of sync with the input after a while in the Holistic Scene

[aadityasp]

trafficstars

Plugin Version or Commit ID

v0.10.1

Unity Version

2021.3.4f1

Your Host OS

macOS Monterey 12.4

Target Platform

Windows Standalone, Android

Target Device

Any Android device

[Windows Only] Visual Studio C++ and Windows SDK Version

No response

[Linux Only] GCC/G++ and GLIBC Version

No response

[Android Only] Android Build Tools and NDK Version

Build Tools: 30.0.2 NDK: 21.4.7075529

[iOS Only] XCode Version

No response

Build Command

python build.py build --desktop cpu --android arm64 --ios arm64 -vv

Used the prebuilt package

Bug Description

Hi, When I try to build the holistic scene, it runs fine in SYNC mode on all platforms such as windows, mac, ios but when deployed to android, the running mode automatically returns to ASYNC mode after some time.(Could it be because of device performance?). I believe this was not done intentionally, because in #661 you said such a thing is not possible. Is there a way to fix this so that the Running mode is always in SYNC mode irrespective of low fps?

I appreciate your help, Thanks!

Steps to Reproduce the Bug

Build Holistic scene to android and change the running mode to SYNC mode and after sometime (probably a minute), you will observe a lag in the segmentation.(goes back to ASYNC)

Log

Was unable to get the android log

Screenshot/Video

https://user-images.githubusercontent.com/14892793/185993390-409d2a6f-2344-4d0b-bdd3-10fc5d8c251b.mp4

from #661

Initially the running mode is Async and you can see a bit of lag in the segmentation.. But After few seconds at 0:40 it goes back to Async mode. (Even though it doesn't explicitly say it's in async mode in the Settings menu). Later when I chose Sync option and saved it, it goes back to being Sync without any lag in segmentation.

Additional Context

No response

[homuler]

Again, it is impossible for the sync mode to switch to async mode on its own. You mean that the output becomes out of sync with the input after a while, right?

[aadityasp]

Yes, the segmentation in the output becomes out of sync with input after a while.

[aadityasp]

Hi @homuler, Do you have any leads on what could be causing this bug?

[homuler]

Will you investigate what is happening when the output gets out of sync with the input? Maybe logging the output value of each stream will be helpful (see https://github.com/homuler/MediaPipeUnityPlugin/issues/642).

In addition, could you check if

1. the queue size is larger than 1? Try inserting UnityEngine.Debug.Log($"{stream} size: {poller.QueueSize()}"); before the below line. https://github.com/homuler/MediaPipeUnityPlugin/blob/d2608ea69676db2a798402a42e00abb9684d1252/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Unity/OutputStream.cs#L326
2. the patch in https://github.com/homuler/MediaPipeUnityPlugin/issues/642#issuecomment-1213648617 fixes your issue?
   • Note that you need to build the library again.

[fengkan]

Will you investigate what is happening when the output gets out of sync with the input? Maybe logging the output value of each stream will be helpful (see #642).

In addition, could you check if

1. the queue size is larger than 1? Try inserting UnityEngine.Debug.Log($"{stream} size: {poller.QueueSize()}"); before the below line. https://github.com/homuler/MediaPipeUnityPlugin/blob/d2608ea69676db2a798402a42e00abb9684d1252/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Unity/OutputStream.cs#L326

2. the patch in poseLandmarksStream sometimes generating an empty packet when observeTimestampBounds = true #642 (comment) fixes your issue?

   • Note that you need to build the library again.

Hi, I am using the Holistic graph, adding the debug line makes something below,

pose_roi size: 0 pose_detection size: 0 pose_landmarks size: 2 face_landmarks size: 1 left_hand_landmarks size: 0 right_hand_landmarks size: 0 pose_world_landmarks size: 2 segmentation_mask size: 1 pose_roi size: 2 pose_detection size: 0 pose_landmarks size: 1 face_landmarks size: 1 left_hand_landmarks size: 0 right_hand_landmarks size: 0 pose_world_landmarks size: 1 segmentation_mask size: 1

Is this output supposed to be correct? What should I check if I add the patch? Thank you.

[homuler]

@fengkan

Is this output supposed to be correct?

I think pose_landmarks is out of sync with the input image (maybe that's why you're posting on this thread).

What should I check if I add the patch?

After applying the patch, please build the native libraries and run the scene. If the image and output are synchronized, then the patch will fix this issue.

[fengkan]

Thank you for the reply, I thought I would get a notification when it's replied.

After applying the patch, please build the native libraries and run the scene. If the image and output are synchronized, then the patch will fix this issue.

I apply the changes and build the libraries again, but the output is basically the same. I am not sure whether I have applied the changes successfully, is there any way I can make sure the libraries really got updated?

Thank you.

[fengkan]

I just modified the files in the _bzl folder according to the mediapipe_debug.diff and run the build cmd, is this correct?

[homuler]

Probably, it's not correct, since bazel can overwrite these files. You need to save the patch and apply it (git apply).

However, unfortunately, the patch I've shared is stale and cannot be applied to the latest version of the plugin.

[fengkan]

Probably, it's not correct, since bazel can overwrite these files. You need to save the patch and apply it (git apply).

However, unfortunately, the patch I've shared is stale and cannot be applied to the latest version of the plugin.

Thank you, haven't tried that yet. And I just wanted to say it seems to be fine in the latest version. I have a video to reproduce the problem, it's OK now.

[homuler]

@fengkan Thank you for reporting it!