MediapipeFaceMeshIOSLibrary icon indicating copy to clipboard operation
MediapipeFaceMeshIOSLibrary copied to clipboard

verified publisher icon swittk
→

Metadata

Building of Mediapipe Facemesh into an easily usable iOS XCFramework

layout: default title: Home nav_order: 1

This is a fork of MediaPipe to demonstrate building MediaPipe as a Framework for iOS, in this case, the FaceMesh model

  • This builds an XCFramework into ./frameworkbuild/FaceMeshIOSLibFramework/xcframework
    • The XCFramework contains both arm64 and x86_64 (iOS Simulator) parts, so you can use this on both real devices and on the iOS simulator
  • I've created the Objective-C file for the framework in //mediapipe/examples/ios/facemeshioslib

Usage

Building

  • prerequisites
    • You need to have Google's Bazel installed. Personally I install via node (npm install -g bazel).
  • run ./BUILD_FACE_MESH_XCFRAMEWORK.sh, the resulting framework should then appear in ./frameworkbuild/FaceMeshIOSLibFramework/xcframework/FaceMeshIOSLibFramework.xcframework
    • Copy the framework and use it in your projects. You're welcome.
    • Framework usage : #import <FaceMeshIOSLibFramework/FaceMeshIOSLibFramework.h>
      • only one class : FaceMeshIOSLib
        • delegate callback gives you an array of detected faces (But there's only one face configured in my graph.. so there's at most length 1)
          • each face is an array of 468 FaceMeshIOSLibFaceLandmarkPoint points (x,y,z)

Resources

  • I couldn't have done this without these sources
    • This great example : https://powderapp.medium.com/mediapipe-tutorial-find-memes-that-match-your-facial-expression-9bf598da98c0
      • Mentioned from issue here
    • Mediapipe graph visualizer : https://viz.mediapipe.dev
    • Understanding mediapipe graphs/calculators : I find it great to start here https://google.github.io/mediapipe/framework_concepts/calculators.html
    • This Stackoverflow thread for pointing out how to create XCFrameworks
      • I tried using good ol' lipo -create ... -output ... with the binaries and putting it in a normal .framework. XCode 12.4 refuses to build with the error Building for iOS Simulator, but the linked and embedded framework was built for iOS + iOS Simulator. It seemed not so long ago that I last used the lipo method. Things change so fast.

MediaPipe

Live ML anywhere

MediaPipe offers cross-platform, customizable ML solutions for live and streaming media.

accelerated.png cross_platform.png
End-to-End acceleration: Built-in fast ML inference and processing accelerated even on common hardware Build once, deploy anywhere: Unified solution works across Android, iOS, desktop/cloud, web and IoT
ready_to_use.png open_source.png
Ready-to-use solutions: Cutting-edge ML solutions demonstrating full power of the framework Free and open source: Framework and solutions both under Apache 2.0, fully extensible and customizable

ML solutions in MediaPipe

Face Detection Face Mesh Iris Hands Pose Holistic
face_detection face_mesh iris hand pose hair_segmentation
Hair Segmentation Object Detection Box Tracking Instant Motion Tracking Objectron KNIFT
hair_segmentation object_detection box_tracking instant_motion_tracking objectron knift
Android iOS C++ Python JS Coral
Face Detection ✅ ✅ ✅ ✅ ✅ ✅
Face Mesh ✅ ✅ ✅ ✅ ✅
Iris ✅ ✅ ✅
Hands ✅ ✅ ✅ ✅ ✅
Pose ✅ ✅ ✅ ✅ ✅
Holistic ✅ ✅ ✅ ✅ ✅
Selfie Segmentation ✅ ✅ ✅ ✅ ✅
Hair Segmentation ✅ ✅
Object Detection ✅ ✅ ✅ ✅
Box Tracking ✅ ✅ ✅
Instant Motion Tracking ✅
Objectron ✅ ✅ ✅ ✅
KNIFT ✅
AutoFlip ✅
MediaSequence ✅
YouTube 8M ✅

See also MediaPipe Models and Model Cards for ML models released in MediaPipe.

Getting started

To start using MediaPipe solutions with only a few lines code, see example code and demos in MediaPipe in Python and MediaPipe in JavaScript.

To use MediaPipe in C++, Android and iOS, which allow further customization of the solutions as well as building your own, learn how to install MediaPipe and start building example applications in C++, Android and iOS.

The source code is hosted in the MediaPipe Github repository, and you can run code search using Google Open Source Code Search.

Publications

Videos

Events

Community

Alpha disclaimer

MediaPipe is currently in alpha at v0.7. We may be still making breaking API changes and expect to get to stable APIs by v1.0.

Contributing

We welcome contributions. Please follow these guidelines.

We use GitHub issues for tracking requests and bugs. Please post questions to the MediaPipe Stack Overflow with a mediapipe tag.

← Metadata

24
Stars
8
Forks
Watchers

Owner

verified publisher icon swittk

Metadata

Building of Mediapipe Facemesh into an easily usable iOS XCFramework

Back