Deep-Live-Cam icon indicating copy to clipboard operation
Deep-Live-Cam copied to clipboard

Published 20 hours ago • verified publisher icon hacksider

Failing to run on Apple Silicon Mac M3

Open betimd opened this issue 9 months ago • 20 comments

Hey there,

Crazy interesting project. Kudos to the team.

Trying to run dev version on a Mac M3 Apple Silicon and running into issue on having live cam.

Appreciate your help 🙏🏻

Logs:

2025-02-23 14:37:22.815 Python[6622:12520853] +[IMKClient subclass]: chose IMKClient_Modern
2025-02-23 14:37:22.815 Python[6622:12520853] +[IMKInputSession subclass]: chose IMKInputSession_Modern
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/betim/.insightface/models/buffalo_l/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/betim/.insightface/models/buffalo_l/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/betim/.insightface/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/betim/.insightface/models/buffalo_l/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/betim/.insightface/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)
Frame processor face_enhancer not found

Waiting for your assistance.

betimd avatar Feb 23 '25 13:02 betimd

same issue

thuytony avatar Feb 25 '25 19:02 thuytony

same issue with message below. FPS equal to 1 when live.

2025-03-02 22:00:45.091 python[23799:1999625] +[IMKInputSession subclass]: chose IMKInputSession_Modern
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/zhiya.pj/.insightface/models/buffalo_l/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/zhiya.pj/.insightface/models/buffalo_l/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/zhiya.pj/.insightface/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/zhiya.pj/.insightface/models/buffalo_l/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/zhiya.pj/.insightface/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
inswapper-shape: [1, 3, 128, 128]```

the gpu usage is 5% when I use mactop to check gpu usage. It's look like coreml didn't work

backnewage avatar Mar 02 '25 14:03 backnewage

try installing the right dependencies

MatteoFari avatar Mar 13 '25 09:03 MatteoFari

What dependencies, how to find right ones? Can you be more specific please?

betimd avatar Mar 13 '25 23:03 betimd

these are the one I run it on:

Name Version Build Channel

absl-py 2.1.0 pypi_0 pypi albucore 0.0.23 pypi_0 pypi albumentations 2.0.5 pypi_0 pypi annotated-types 0.7.0 pypi_0 pypi astunparse 1.6.3 pypi_0 pypi bzip2 1.0.8 h80987f9_6
ca-certificates 2025.2.25 hca03da5_0
certifi 2025.1.31 pypi_0 pypi charset-normalizer 3.4.1 pypi_0 pypi coloredlogs 15.0.1 pypi_0 pypi contourpy 1.3.1 pypi_0 pypi customtkinter 5.2.2 pypi_0 pypi cv2-enumerate-cameras 1.1.18.3 pypi_0 pypi cycler 0.12.1 pypi_0 pypi cython 3.0.12 pypi_0 pypi darkdetect 0.8.0 pypi_0 pypi easydict 1.13 pypi_0 pypi filelock 3.17.0 pypi_0 pypi flatbuffers 25.2.10 pypi_0 pypi fonttools 4.56.0 pypi_0 pypi fsspec 2025.3.0 pypi_0 pypi gast 0.6.0 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.71.0 pypi_0 pypi h5py 3.13.0 pypi_0 pypi humanfriendly 10.0 pypi_0 pypi idna 3.10 pypi_0 pypi imageio 2.37.0 pypi_0 pypi insightface 0.7.3 pypi_0 pypi jinja2 3.1.6 pypi_0 pypi joblib 1.4.2 pypi_0 pypi keras 3.9.0 pypi_0 pypi kiwisolver 1.4.8 pypi_0 pypi lazy-loader 0.4 pypi_0 pypi libclang 18.1.1 pypi_0 pypi libffi 3.4.4 hca03da5_1
markdown 3.7 pypi_0 pypi markdown-it-py 3.0.0 pypi_0 pypi markupsafe 3.0.2 pypi_0 pypi matplotlib 3.10.1 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi ml-dtypes 0.5.1 pypi_0 pypi mpmath 1.3.0 pypi_0 pypi namex 0.0.8 pypi_0 pypi ncurses 6.4 h313beb8_0
networkx 3.4.2 pypi_0 pypi numpy 1.26.4 pypi_0 pypi onnx 1.17.0 pypi_0 pypi onnxruntime-genai 0.6.0 pypi_0 pypi onnxruntime-silicon 1.16.0 pypi_0 pypi opencv-python 4.11.0.86 pypi_0 pypi opencv-python-headless 4.11.0.86 pypi_0 pypi openssl 3.0.16 h02f6b3c_0
opt-einsum 3.4.0 pypi_0 pypi optree 0.14.1 pypi_0 pypi packaging 24.2 pypi_0 pypi pillow 11.1.0 pypi_0 pypi pip 25.0 py310hca03da5_0
prettytable 3.15.1 pypi_0 pypi protobuf 5.29.3 pypi_0 pypi pydantic 2.10.6 pypi_0 pypi pydantic-core 2.27.2 pypi_0 pypi pygments 2.19.1 pypi_0 pypi pyparsing 3.2.1 pypi_0 pypi python 3.10.16 hb885b13_1
python-dateutil 2.9.0.post0 pypi_0 pypi pyyaml 6.0.2 pypi_0 pypi readline 8.2 h1a28f6b_0
requests 2.32.3 pypi_0 pypi rich 13.9.4 pypi_0 pypi scikit-image 0.25.2 pypi_0 pypi scikit-learn 1.6.1 pypi_0 pypi scipy 1.15.2 pypi_0 pypi setuptools 75.8.0 py310hca03da5_0
simsimd 6.2.1 pypi_0 pypi six 1.17.0 pypi_0 pypi sqlite 3.45.3 h80987f9_0
stringzilla 3.12.3 pypi_0 pypi sympy 1.13.1 pypi_0 pypi tensorboard 2.19.0 pypi_0 pypi tensorboard-data-server 0.7.2 pypi_0 pypi tensorflow 2.19.0 pypi_0 pypi tensorflow-io-gcs-filesystem 0.37.1 pypi_0 pypi termcolor 2.5.0 pypi_0 pypi threadpoolctl 3.5.0 pypi_0 pypi tifffile 2025.2.18 pypi_0 pypi tk 8.6.14 h6ba3021_0
torch 2.6.0 pypi_0 pypi torchaudio 2.6.0 pypi_0 pypi torchvision 0.21.0 pypi_0 pypi tqdm 4.67.1 pypi_0 pypi typing-extensions 4.12.2 pypi_0 pypi tzdata 2025a h04d1e81_0
urllib3 2.3.0 pypi_0 pypi wcwidth 0.2.13 pypi_0 pypi werkzeug 3.1.3 pypi_0 pypi wheel 0.45.1 py310hca03da5_0
wrapt 1.17.2 pypi_0 pypi xz 5.6.4 h80987f9_1
zlib 1.2.13 h18a0788_1

MatteoFari avatar Mar 14 '25 09:03 MatteoFari

These are all installed when you do pip install, right? Did you installed separately deps? Where did you get reqs.txt or any list of deps?

betimd avatar Mar 14 '25 12:03 betimd

by running the code with the default requirements i ran into some errors, I checked them, installed the right dep and after 2-3 trial and errors it ran fine

MatteoFari avatar Mar 14 '25 14:03 MatteoFari

Could you make a reqs file with proper deps and right versioning and share it here with us please?

betimd avatar Mar 14 '25 14:03 betimd

guys, you must to install the right versions, otherwise will not work! here, I create one PR that might bring all the necessary steps. https://github.com/hacksider/Deep-Live-Cam/pull/990

locally I running Python 3.10.16

wpoPR avatar Mar 21 '25 17:03 wpoPR

I followed the steps from https://github.com/hacksider/Deep-Live-Cam/pull/990 exactly and even after 2-3 tries it works but I only get 1.5 fps with M1 Max.

Can't see anything obviously wrong from this but looks like it is still CPU bound as GFX cores are nearly idle when running.

(venv) jacob@jcmbp Deep-Live-Cam % python3.10 run.py --execution-provider coreml
Frame processor face_enhancer not found
OpenCV: out device of bound (0-0): 1
OpenCV: camera failed to properly initialize!
[03/22 12:04:11.510713][info][15195500][Context.cpp:69] Context created with config: default config!
[03/22 12:04:11.510739][info][15195500][Context.cpp:74] Context work_dir=/Users/jacob/REPOS/Deep-Live-Cam
[03/22 12:04:11.510741][info][15195500][Context.cpp:77]         - SDK version: 1.9.4
[03/22 12:04:11.510742][info][15195500][Context.cpp:78]         - SDK stage version: main
[03/22 12:04:11.510745][info][15195500][Context.cpp:82] get config EnumerateNetDevice:false
[03/22 12:04:11.510747][info][15195500][MacPal.cpp:36] createObPal: create MacPal!
[03/22 12:04:11.510997][info][15195500][MacPal.cpp:104] Create PollingDeviceWatcher!
[03/22 12:04:11.511007][info][15195500][DeviceManager.cpp:15] Current found device(s): (0)
[03/22 12:04:11.511011][info][15195500][Pipeline.cpp:15] Try to create pipeline with default device.
[03/22 12:04:11.511013][warning][15195500][ObException.cpp:5] No device found, fail to create pipeline!
[03/22 12:04:11.511310][info][15195500][Context.cpp:90] Context destroyed
[ WARN:[email protected]] global cap.cpp:323 open VIDEOIO(OBSENSOR): raised unknown C++ exception!


OpenCV: out device of bound (0-0): 2
OpenCV: camera failed to properly initialize!
[03/22 12:04:11.525560][info][15195500][Context.cpp:69] Context created with config: default config!
[03/22 12:04:11.525577][info][15195500][Context.cpp:74] Context work_dir=/Users/jacob/REPOS/Deep-Live-Cam
[03/22 12:04:11.525578][info][15195500][Context.cpp:77]         - SDK version: 1.9.4
[03/22 12:04:11.525580][info][15195500][Context.cpp:78]         - SDK stage version: main
[03/22 12:04:11.525583][info][15195500][Context.cpp:82] get config EnumerateNetDevice:false
[03/22 12:04:11.525585][info][15195500][MacPal.cpp:36] createObPal: create MacPal!
[03/22 12:04:11.525731][info][15195500][MacPal.cpp:104] Create PollingDeviceWatcher!
[03/22 12:04:11.525737][info][15195500][DeviceManager.cpp:15] Current found device(s): (0)
[03/22 12:04:11.525740][info][15195500][Pipeline.cpp:15] Try to create pipeline with default device.
[03/22 12:04:11.525742][warning][15195500][ObException.cpp:5] No device found, fail to create pipeline!
[03/22 12:04:11.525887][info][15195500][Context.cpp:90] Context destroyed
[ WARN:[email protected]] global cap.cpp:323 open VIDEOIO(OBSENSOR): raised unknown C++ exception!


2025-03-22 12:04:25.520 Python[62047:15195500] +[IMKClient subclass]: chose IMKClient_Modern
2025-03-22 12:04:25.520 Python[62047:15195500] +[IMKInputSession subclass]: chose IMKInputSession_Modern
Frame processor face_enhancer not found
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/jacob/.insightface/models/buffalo_l/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/jacob/.insightface/models/buffalo_l/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/jacob/.insightface/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/jacob/.insightface/models/buffalo_l/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
find model: /Users/jacob/.insightface/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)
ERROR:root:Face detection failed for target or source.
Applied providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'CoreMLExecutionProvider': {}}
inswapper-shape: [1, 3, 128, 128]

jacobpretorius avatar Mar 22 '25 12:03 jacobpretorius

yeah because it's not optimized for cpu only yet

MatteoFari avatar Mar 22 '25 16:03 MatteoFari

@jacobpretorius yep, same here! soon they might have good stuff for mac :)

wpoPR avatar Mar 24 '25 12:03 wpoPR

I had the same problem, eventually I realized unchecking the face enhancer let it work.

justinmorel avatar Apr 04 '25 02:04 justinmorel

I'm on M4 Pro, I had the same issue with it only running at around 1 fps until I updated onnxruntime_silicon to the latest version (1.16.3) and switched to the full-size inswapper_128.onnx (over 500MB) instead of inswapper_128_fp16.onnx (about 200MB) that they recommended. Now it is giving me 5-6 fps. Note that CoreML seems to run this model on ANE instead of GPU. I got about 20% of ANE usage while running it.

gywghhb avatar Apr 05 '25 23:04 gywghhb

I'm on M4 Pro, I had the same issue with it only running at around 1 fps until I updated onnxruntime_silicon to the latest version (1.16.3) and switched to the full-size inswapper_128.onnx (over 500MB) instead of inswapper_128_fp16.onnx (about 200MB) that they recommended. Now it is giving me 5-6 fps. Note that CoreML seems to run this model on ANE instead of GPU. I got about 20% of ANE usage while running it.

Ok this improved my FPS from 0.3FPS and brought it up to 4 FPS on Mac M1Pro

phillips321 avatar Apr 09 '25 08:04 phillips321

I'm on M4 Pro, I had the same issue with it only running at around 1 fps until I updated onnxruntime_silicon to the latest version (1.16.3) and switched to the full-size inswapper_128.onnx (over 500MB) instead of inswapper_128_fp16.onnx (about 200MB) that they recommended. Now it is giving me 5-6 fps. Note that CoreML seems to run this model on ANE instead of GPU. I got about 20% of ANE usage while running it.

Any idea how to run this on the GPUs instead of ANE?

phillips321 avatar Apr 10 '25 14:04 phillips321

How can I install it

AMAZINGGODDID avatar Apr 11 '25 19:04 AMAZINGGODDID

using inswapper_128.onnx and installing the latest onnxruntime_silicon 1.16.3 got me up to about ~5 FPS on an Air M4 without Face Enhancer. Face enhancer dropped it down to ~1 FPS, but didn't seem to improve quality meaningfully. Thanks @gywghhb good find.

Would love to get this working on the GPU

josephtutera avatar May 04 '25 18:05 josephtutera

Just a heads up, the package onnxruntime_silicon should NOT be installed on Apple silicon macs, since it's outdated

⚠️ The official ONNX Runtime now includes arm64 binaries for MacOS as well with Core ML support. Please use the official wheel package as this repository is no longer needed.

Source: https://github.com/cansik/onnxruntime-silicon

✅ Instead

pip uninstall onnxruntime-silicon

pip install onnxruntime

avarayr avatar May 05 '25 18:05 avarayr

Just a heads up, the package onnxruntime_silicon should NOT be installed on Apple silicon macs, since it's outdated

⚠️ The official ONNX Runtime now includes arm64 binaries for MacOS as well with Core ML support. Please use the official wheel package as this repository is no longer needed.

Source: cansik/onnxruntime-silicon

✅ Instead

pip uninstall onnxruntime-silicon

pip install onnxruntime

Returns this on M1:

AttributeError: module 'onnxruntime' has no attribute 'InferenceSession'

moritzschmitz-oviva avatar May 13 '25 08:05 moritzschmitz-oviva