NormalizedLandmark returns x,y,z coordinate values outside [0.0, 1.0]

[ErikValle2]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

Ubuntu 22.04.4 LTS

Mobile device if the issue happens on mobile device

No response

Browser and version if the issue happens on browser

No response

Programming Language and version

Python

MediaPipe version

0.10.13

Bazel version

No response

Solution

Pose

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

No response

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

NormalizedLandmark gives some values out of boundaries. The only difference between pose_landmarks and NormalizedLandmark is that the second option contains visibility and presence scores. I tried with latest/pose_landmarker_heavy.task and 1/pose_landmarker_heavy.task, they give the same results

Describe the expected behaviour

According to the documentation, Normalized Landmark represents a point in 3D space with x, y, z coordinates. x and y are normalized to [0.0, 1.0] by the image width and height respectively. z represents the landmark depth, and the smaller the value the closer the landmark is to the camera. I understand Mediapipe tries to make up the "invisible" joints, but wouldn't be better if it simply does not include them?

Standalone code/steps you may have used to try to get what you need

#!wget -O pose_landmarker.task -q https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_heavy/float16/latest/pose_landmarker_heavy.task

from mediapipe import solutions
from mediapipe.framework.formats import landmark_pb2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
import cv2
import numpy as np

# STEP 1: Create an PoseLandmarker object.
base_options = python.BaseOptions(model_asset_path='pose_landmarker.task')
options = vision.PoseLandmarkerOptions(
    base_options=base_options,
    output_segmentation_masks=True)
detector = vision.PoseLandmarker.create_from_options(options)

# STEP 2: Load input images
frame = mp.Image.create_from_file('webcam1_erik3_cal_50.jpg') #RGB image, 638x478
detection_result = detector.detect(frame)

# Function to count landmarks
def count_landmarks(landmarks):
    return [sum(1 for lm in landmarks if 0 < lm.x > 1), sum(1 for lm in landmarks if 0 < lm.y > 1)]

#analysis
pose_lm = count_landmarks(detection_result.pose_landmarks[0])
pose_norm_lm = count_landmarks([landmark_pb2.NormalizedLandmark(x=landmark.x, y=landmark.y, z=landmark.z) for landmark in detection_result.pose_landmarks[0]])

pose_landmarks_proto = landmark_pb2.NormalizedLandmarkList()
pose_landmarks_proto.landmark.extend([landmark_pb2.NormalizedLandmark(x=landmark.x, y=landmark.y, z=landmark.z) for landmark in detection_result.pose_landmarks[0]])
pose_proto_lm = count_landmarks(pose_landmarks_proto.landmark)

print(f"# of pose Landmarks out of boundaries: {pose_lm}")
print(f"# of normalized pose landmarks out of boundaries: {pose_norm_lm}")
print(f"# of proto pose landmarks out of boundaries: {pose_proto_lm}")

Other info / Complete Logs

# of pose Landmarks out of boundaries: [0, 19]
# of normalized pose landmarks out of boundaries: [0, 19]
# of proto pose landmarks out of boundaries: [0, 19]
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1718176661.923482   43351 task_runner.cc:85] GPU suport is not available: INTERNAL: ; RET_CHECK failure (mediapipe/gpu/gl_context_egl.cc:77) display != EGL_NO_DISPLAYeglGetDisplay() returned error 0x300c
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
/usr/local/lib/python3.11/site-packages/google/protobuf/symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon.
  warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please '

[aliakseibrown]

Can you send the picture 'webcam1_erik3_cal_50.jpg' to test?

[ayushgdev]

@ErikValle2 Thanks for bringing this to our attention. We are able to reproduce the issue at our end. We will investigate this internally and find the root cause.

[kuaashish]

Hi @ErikValle2,

Could you please verify if the issue is still present in the most recent version 0.10.18 and let us know the status?

Thank you!!

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No