Can you expand this masterpiece to support multithreaded/multi core parallel processing?

[LeMetamax]

Feature Description

Just like the unity job system works, splitting code into multiple pieces to be executed by different/all cpu cores thereby boosting performance. I think it'll make a huge difference if mediapipe is able to do the same thing. Maybe not necessarily utilizing the job system and burst compiler. But using some other technique maybe the new Awaitable system from Unity 2023.x or System.Threading...

Current Behaviour/State

Right now, the tool is performing efficiently on most PCs and other devices with excellent processing capabilities. But it seems to be running on the main thread alone/mostly. Especially when you run the scenes on sync mode with segmentation and on android phones, the performance drastically reduces.

Additional Context

No response

[homuler]

Please see https://github.com/homuler/MediaPipeUnityPlugin/issues/1052#issuecomment-1879599308. Could you please tell me what prevents you from running this plugin on multi-thread?

But it seems to be running on the main thread alone/mostly.

MediaPipe itself runs on other threads.

Especially when you run the scenes on sync mode with segmentation and on android phones, the performance drastically reduces.

That's because it waits for the Task completion on the main thread. https://github.com/homuler/MediaPipeUnityPlugin/blob/9e2d60132659ad7c24cc7442a02c5de63d37975a/Assets/MediaPipeUnity/Samples/Scenes/Pose%20Tracking/PoseTrackingSolution.cs#L90-L91

Of course, you wouldn't write such code in a real application.

Also, note that the above code (code under Assets/) is merely a sample. It doesn't mean you cannot run the CalculatorGraph on other thread than the main thread.

[LeMetamax]

Thanks for responding. I'll take a better look at this and get back.

[LeMetamax]

I was looking at an older version initially. Anyways thanks for pointing this out. And yes, it does indeed run on a background thread.

Of course, you wouldn't write such code in a real application.

I guess this is also the reason why on 0.14.1 sync mode, the segmentation and landmarks are very choppy/laggy even though my fps counter shows the app running at a steady 30FPS. I tested this theory by updating to 2023.2.8f1 to use the Awaitable API and then spending hours converting the tasks and coroutine on the Holistic Tracking Graph/Solution and the OutputStream.

Since we shouldn't use it in a real application, can you please tell me...what do you suggest? I thought of the JobSystem but we'll likely run into errors given that we won't be passing value types.

[homuler]

How about using UniTask?

I'm not sure where you're feeling the difficulty, but I think either:

• Instead of waiting for inference to finish, receive the inference results via a callback as in the async mode.
• Run the MediaPipe-related processes on a separate thread (using UniTask)

should suffice.