Please help with getting landmark coordinate from the Hand Tracking sample

[flaxias]

Plugin Version or Commit ID

v0.14.0

Unity Version

2022.3.16f1

Your Host OS

Windows 11

Target Platform

Android

Description

I'm trying to attach a cube to the index finger, but the movement/placement is not the same as the finger position on screen. This is my current code. I just modified HandTrackingSolution.cs and added this.

foreach (var landmarks in landmarksList)
    {
        var indexFinger = landmarks.Landmark[8];

        // Convert indexFinger coordinates to local coordinates of the _screen
        Vector2 localPoint;
        RectTransformUtility.ScreenPointToLocalPointInRectangle(_screen.rectTransform, screenRect.GetPoint(indexFinger), null, out localPoint);

        // Normalize the local coordinates based on the screen size
        Vector2 normalizedLocalPoint = new Vector2(localPoint.x / screenRect.width, localPoint.y / screenRect.height);    

        cube.transform.position = normalizedLocalPoint;
     }   

Can anyone please help? Thank you

[homuler]

Please read https://github.com/homuler/MediaPipeUnityPlugin/wiki/Getting-Started#coordinate-system and https://github.com/homuler/MediaPipeUnityPlugin/wiki/Getting-Started#get-landmarks first.

[flaxias]

I have, but if I directly used:

var indexFinger = landmarks.Landmark[8];
        var position = screenRect.GetPoint(indexFinger);
        cube.transform.position = position;

Then the cube position is not covering the finger in the UI image. Sorry if this is a newbie question, I'm new to Unity.

[flaxias]

A little help please? Anyone? At least point me to the right direction

[homuler]

Since it depends on the app, I'm not sure, but is it correct to use localPosition?

[bbq82828]

This is my method for your reference：

camera.ScreenToWorldPoint(new Vector3(vector3s[index].x * Screen.width,vector3s[index].y * Screen.height, vector3s[index].z ))

You need to convert the obtained landmark into world coordinates

[flaxias]

This is my method for your reference：

camera.ScreenToWorldPoint(new Vector3(vector3s[index].x * Screen.width,vector3s[index].y * Screen.height, vector3s[index].z ))

You need to convert the obtained landmark into world coordinates

Hi, thank you for your help. I tried using your method, but it still have the same issue as my previous method. The movement is mostly correct in the X direction, but Y movement is off. Especially if I scaled the canvas. I think my issue is similar to this one: https://github.com/homuler/MediaPipeUnityPlugin/issues/1029

I think it's due to the scaling. I'm using the example project with the Auto Fit script in the Annotatable Screen. Here's a video I recorded to better show the issue: Hand tracking position problem.

I want to deploy this to android, which have different screen sizes, and I want the position to be accurate across all of them despite the screen ratio. Is this possible?

Thank you.

[homuler]

With this plugin, you can get the hand positions within images. However, what you do with that data is up to you. Therefore, I'm open to questions up to the point of retrieving the data, but if you're unsure about how to proceed with what you want to achieve after obtaining the data (especially how to use Unity), please ask elsewhere.