supervision
supervision copied to clipboard
roboflow
Add sv.Keypoints method
Description
This PR adds the sv.Keypoints detection method. This method can accept keypoints in (N, M, 2) (for x, y points) or (N, M, 3) formats (for x, y, z points).
An sv.PoseAnnotator method is added for plotting poses.
sv.Keypoints.from_ultralytics() is supported for loading keypoints from ultralytics pose estimation models.
Type of change
- [X] New feature (non-breaking change which adds functionality)
How has this change been tested, please provide a testcase or example of how you tested the change?
This change was tested using the following code:
from ultralytics import YOLO
import supervision as sv
import cv2
# test empty
print(sv.Keypoints.empty())
model = YOLO('yolov8n-pose.pt')
results = model('https://ultralytics.com/images/bus.jpg')
keypoints = sv.Keypoints.from_ultralytics(results)
image = cv2.imread("bus.jpg")
pose_annotator = sv.PoseAnnotator()
annotated_frame = pose_annotator.annotate(
scene=image,
keypoints=keypoints
)
sv.plot_image(annotated_frame)
Any specific deployment considerations
N/A
Docs
I will add documentation for sv.Keypoint when we finalize the API.
Here is an example of an image where keypoints have been plotted with the sv.PoseAnnotator method:
Some keypoint and face landmark models also provide bounding boxes for the people or faces detected. Should these be part of sv.Keypoint()? If so, how should this information be included in the object?
Some keypoint and face landmark models also provide bounding boxes for the people or faces detected. Should these be part of
sv.Keypoint()? If so, how should this information be included in the object?
I was also playing with this last week to complete and I was just focused on KeyPoints Results
I have just added a sv.Keypoint.from_mediapipe() loader. Test code:
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
base_options = python.BaseOptions(model_asset_path='pose_landmarker_heavy.task')
options = vision.PoseLandmarkerOptions(
base_options=base_options,
output_segmentation_masks=True)
detector = vision.PoseLandmarker.create_from_options(options)
image = mp.Image.create_from_file("bus.jpg")
detection_result = detector.detect(image)
import supervision as sv
import cv2
keypoints = sv.Keypoints.from_mediapipe(detection_result)
image = cv2.imread("bus.jpg")
pose_annotator = sv.PoseAnnotator()
annotated_frame = pose_annotator.annotate(
scene=image,
keypoints=keypoints
)
sv.plot_image(annotated_frame)
Follow the instructions in the MediaPipe landmark notebook to install MediaPipe and download the model weights.
@capjamesg, notice that your pre-commit is failing. Also, please create colab where we can easily test proposed API.
like I would make this configurable - obviously skeleton would require from user to pass the list of edges as input
I'd say we should support most common Pose formats, and allow to pass custom edges.
I found issues in data loaders of sv.Keypoints when no detections are present or objects are detected (this one in from_ultralytics only).
Feel free to include it, if you find it useful. PR: #948
I found issues in data loaders of
sv.Keypointswhen no detections are present or objects are detected (this one infrom_ultralyticsonly).Feel free to include it, if you find it useful. PR: #948
I wasn't using this PR when I was working on it. I also made my work so If you don't mind I will take look and close yours and send mine with docs and other stuff
I found issues in data loaders of
sv.Keypointswhen no detections are present or objects are detected (this one infrom_ultralyticsonly). Feel free to include it, if you find it useful. PR: #948I wasn't using this PR when I was working on it. I also made my work so If you don't mind I will take look and close yours and send mine with docs and other stuff
But Thank you for your attempt as well.
I'm closing this PR as KeyPoints support was added in https://github.com/roboflow/supervision/pull/1128