How to calculate the palm distance from the camera

[ShonubiSamuel]

Plugin Version or Commit ID

v0.12.0

Unity Version

2022.3.4f1

Your Host OS

macOS Ventura 13.6.1

Target Platform

Android

Description

Hi, I am currently working on an AR project in Unity, utilizing the hand tracking solution provided by MediaPipe through AR Foundation. My goal is to accurately determine the palm distance from the camera for better interaction with virtual objects.

Currently, I've implemented the hand tracking solution to work with unity arfoundation. However, I'm facing a challenge in ensuring that these annotations are appropriately positioned in 3D space relative to the AR camera. I aim to have the annotations at the correct distance from the camera, allowing users to interact with objects at varying depths using their hands.

To address this issue, I attempted a solution involving placing my hand at different distances from the camera and recording the distance between two specific balls in the hand tracking solution. By analyzing the data, I identified a quadratic pattern and derived the quadratic coefficient to calculate palm distance at runtime. While this approach provided some success, I encountered issues with the annotation not mapping accurately in certain scenarios.

I'm reaching out to seek your guidance. Are there better methods or best practices for accurately determining palm distance in an AR environment using MediaPipe in Unity? I would greatly appreciate any insights or recommendations you may have on improving the accuracy and reliability of palm distance calculations.

Here is a sample of how mine looks like.

https://github.com/homuler/MediaPipeUnityPlugin/assets/63505340/ba9f9bc1-4596-440e-9d7a-a5d683bb5df9

Code to Reproduce the issue

No response

Additional Context

No response

[homuler]

As far as I know, we cannot estimate the depth with MediaPipe, or if we can, the accuracy might not be high. Therefore, I think you either need to do it by yourself (as you've already done) or combine it with a depth camera for better results. For accurate information, please ask within the official MediaPipe community.