Adds a new Pose Detection Frame Processor Plugin to the VisionCamera example app, enabling real-time human pose detection and skeleton visualization directly in the camera feed.

[littlebullGit]

Trying to demonstrate how to integrate ML with the vision camera

Files Created/Updated

• Added Swift implementation (PoseDetectionFrameProcessor.swift ) that uses TensorFlow Lite to detect human poses in camera frames
• Added Objective-C bridge (PoseDetectionFrameProcessor.m ) to register the plugin with React Native
• Created TypeScript interface (PoseDetectionPlugin.ts ) for the native plugin
• AddedPoseSkeletonOverlay.tsx component for rendering the detected pose skeleton
• Integrated pose detection intoCameraPage.tsx with UI controls
• Added TensorFlow Lite models (MoveNet Thunder) in the PoseModels directory
• Updated Xcode project configuration to include the new files and models

Implementation Details

• Uses MoveNet models from TensorFlow for efficient on-device pose detection
• Supports Thunder (more accurate). Testing of Lightning model did not yet good enough confidence
• Processes camera frames in real-time to detect 17 body keypoints
• Handles coordinate transformations between the model, camera frame, and UI
• Renders an overlay with keypoints and connections (skeleton) on top of the camera preview
• Provides confidence scores for each detected keypoint
• Properly handles front/back camera mirroring

Testing

• Tested on iPhone 13, iOS 16.5
• Tested on iPhone 14, iOS 18.0
• Tested with both front and back cameras
• did not have Android implementation as I have no android device This implementation demonstrates how to create complex frame processor plugins that leverage machine learning models while maintaining good performance on device.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
react-native-vision-camera	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 1, 2025 0:55am