(almost) Immediate native crash on iOS

[cavisions]

Plugin Version or Commit ID

v0.14.0, v0.12.0

Unity Version

2021.3.31f

Your Host OS

Windows 11

Target Platform

iOS

Target Device

iPhone XR, iPhone 13 mini, iPad Pro (11‑inch)

[Windows Only] Visual Studio C++ and Windows SDK Version

No response

[Linux Only] GCC/G++ and GLIBC Version

No response

[Android Only] Android Build Tools and NDK Version

No response

[iOS Only] XCode Version

15.1

Build Command

We are using the libraries build by you and available to be downloaded from https://github.com/homuler/MediaPipeUnityPlugin/releases

Bug Description

We receive a native crash several seconds after starting the application that uses your libraries. It is observed in both our application and your samples. In exactly the same conditions the Windows and Android versions are working well. The crash is very easy to reproduce on devices with small RAM amount (e.g. iPhone XR) - it takes less than 1 second to crash when using the Full or Heavy model. On devices with more RAM it works for a while, but when changing to Heavy model it crashes very quickly.

Steps to Reproduce the Bug

1. Start your PoseTracking sample on iOS device
2. Wait for a while (change to a heavier model on devices with more RAM)
3. The sample will crash

Log

In the xcode console there is a warning: WARNING -> applicationDidReceiveMemoryWarning()

This is what we observe while trying to debug the native code:

Screenshot/Video

No response

Additional Context

No response

[homuler]

Is the official sample app running successfully? If it runs, how much difference in memory consumption is there compared to the app that crashes?

[cavisions]

Is the official sample app running successfully? If it runs, how much difference in memory consumption is there compared to the app that crashes?

No - your official sample app also crashes. It works ok for the face tracking (memory consumption is ~420MB), but when pose tracking is used the memory consumption starts rising and when it gets to ~600MB (what takes about 1-2 seconds) the app crashes.

[homuler]

I mean the MediaPipe official sample.

[homuler]

How about disabling segmentation mask?

https://github.com/homuler/MediaPipeUnityPlugin/blob/251ebd350c5e74cfcbc1f37a68d73dfb9a833f95/Assets/MediaPipeUnity/Samples/Scenes/Pose%20Tracking/PoseTrackingGraph.cs#L27

[cavisions]

I mean the MediaPipe official sample.

We have successfully started the sample, but there is no PoseTracking there (and the face detection - it works good exactly like in your samples). There are samples for Android, Python and Web here https://developers.google.com/mediapipe/solutions/vision/pose_landmarker but there is no iOS.

[cavisions]

How about disabling segmentation mask?

https://github.com/homuler/MediaPipeUnityPlugin/blob/251ebd350c5e74cfcbc1f37a68d73dfb9a833f95/Assets/MediaPipeUnity/Samples/Scenes/Pose%20Tracking/PoseTrackingGraph.cs#L27

We'll have a look, but anyhow we strongly need the segmentation mask in our application, so it will not save us :(

[cavisions]

How about disabling segmentation mask?

https://github.com/homuler/MediaPipeUnityPlugin/blob/251ebd350c5e74cfcbc1f37a68d73dfb9a833f95/Assets/MediaPipeUnity/Samples/Scenes/Pose%20Tracking/PoseTrackingGraph.cs#L27

UPDATE: When the segmentation mask is disabled it looks like it is working correctly using about 500MB of memory. Anyhow I'd appreciate any suggestions that will help us have the segmentation mask working, because we use it quite heavily in our app. Thank you in advance!

[homuler]

If there is a way to make it work, I think the following approach would be effective:

1. Upon receiving the segmentation mask (e.g. ImageFrame), use TryReadChannelNormalized to copy the data to a pre-allocated buffer.
2. Promptly dispose of the segmentation mask data (e.g. ImageFrame.Dispose()).

Additionally, reducing the camera resolution could also be effective.

[cavisions]

If there is a way to make it work, I think the following approach would be effective:

1. Upon receiving the segmentation mask (e.g. ImageFrame), use TryReadChannelNormalized to copy the data to a pre-allocated buffer.
2. Promptly dispose of the segmentation mask data (e.g. ImageFrame.Dispose()).

Additionally, reducing the camera resolution could also be effective.

Thank you. We'll try it for sure.

BTW: Is there any timeframe the next release (that uses mediapipe v.0.10.9) is planned?

[homuler]

Although there remain some known bugs, but to be honest, it's fine to release at any time. There are some breaking changes, so I'm trying to include other breaking changes altogether, but it's OK to skip them.

[cavisions]

Although there remain some known bugs, but to be honest, it's fine to release at any time. There are some breaking changes, so I'm trying to include other breaking changes altogether, but it's OK to skip them.

That would make me super excited to play with it!

[cavisions]

If there is a way to make it work, I think the following approach would be effective:

1. Upon receiving the segmentation mask (e.g. ImageFrame), use TryReadChannelNormalized to copy the data to a pre-allocated buffer.
2. Promptly dispose of the segmentation mask data (e.g. ImageFrame.Dispose()).

Additionally, reducing the camera resolution could also be effective.

We've tested all of the suggestions above, but nothing seems to work stable. For sure there is a correlation between resolution and model. The higher resolution and the heavier model is used the quicker it crashes. We have tested also the v.0.14.0 version, but the behavior is exactly the same.

Any additional suggestions would be greatly appreciated. Thank you for all your hard work!

[homuler]

We have tested also the v.0.14.0 version, but the behavior is exactly the same.

Have you tried the PoseLandmarker API?

Any additional suggestions would be greatly appreciated.

I don't know the minimum required memory size to run it, but if it also doesn't work, I think that means you cannot run the Full or Heavy model on the device. By the way, why you don't run the Lite model instead?

[cavisions]

@homuler Do you have any plans to update to Mediapipe v.0.10.14?

[homuler]

I'd like to update the version, but I haven't been able to due to lack of time. At the moment, I'm hoping to get it done within the next few months.