How to properly use Python multiprocessing for reducing delays (hand landmarker)

[astariul]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

Mac Darwin Kernel Version 21.6.0

MediaPipe Tasks SDK version

0.10.9

Task name (e.g. Image classification, Gesture recognition etc.)

hand landmarker

Programming Language and version (e.g. C++, Python, Java)

Python

Describe the actual behavior

I followed the Python guide for hand landmarks detection. It works well, I'm using the LIVE_STREAM mode.

However, because some frames are skipped, it sometimes feel like there is delays between the frame being displayed and the hand landmarks (the landmarks "lags" behind the frame being displayed).

I tried to run the model in VIDEO mode to have blocking calls, and it helps (the landmarks appears exactly at the right place), but at the cost of the video framerate going down... (which is expected)

Describe the expected behaviour

So I'm thinking of how to improve this. I thought about using multiprocessing, but I'm unsure it's the right path.

Because if I have several processes, it means I have several mediapipe detectors. But in VIDEO and LIVE_STREAM, the model relies on the previous hands positions to accelerate the detection in the next frame.
But in the case of multiprocessing, the previous frame (cached by the mediapipe object) might be much older than the actual last frame (if the recent frames were treated by the other process)... So it seems not optimal.

Alternatively, I can force alternating between processes : I send the even frame to the first process, and the odd frame to the second process.

Anyway, I wonder if this is the right approach, or if there is a better way to improve the situation with multiprocessing ? Or something else ?