avatarify-python
avatarify-python copied to clipboard
alievk
Faster (realtime) GPU face_alignment
You can use (or modify face-alignment) yourself to get realtime face landmark with this code: https://github.com/kwea123/VTuber_Unity/tree/master/face_alignment
It's really plug and play, except you'll have to integrate it in your repo (and do some script to automatically download the models).
Unfortunately, its not yet possible to get decent CPU performance with first-order-model because ONNX doesn't yet support grid_sampler operations.
It's quite easy, actually.
pip install onnxruntime
Just put his face_alignment folder somewhere, download and unzip https://github.com/kwea123/VTuber_Unity/releases/latest/download/models.zip to face_alignment/ckpts
Now you just import face_alignment (remember to pip uninstall the official version pip uninstall face_alignment) and do
fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D, flip_input=True,
device=device, use_onnx=True)
If you put his face_alignment folder somewhere outside of your avatarify folder, put this in the last line of your imports:
sys.path.insert(0, "path/to/face_aligment_parent_folder")
import face_alignment
Intel(R) Xeon(R) CPU E3-1575M v5 @ 3.00GHz without CUDA/NVIDIA, I get ~4 FPS. I think this is better than with the published face_alignment; still barely demonstrable.
On Debian, with protobuf 3.11, I had to build/install onnx from source.
As well, setting flip_input=True causes a crash due to numpy / video-input issue; change this to False and it seems alright (video is flipped at some point? flipping video in avatarify still works... ???).
pip3 install numpa, and some other one I forget now.
Last, can probably use v4l2loopback packages shipped with Debian but I did not attempt.
Intel(R) Xeon(R) CPU E3-1575M v5 @ 3.00GHzwithout CUDA/NVIDIA, I get ~4 FPS. I think this is better than with the publishedface_alignment; still barely demonstrable.
I got it wrong. In CPU I get 1.68s per face-alignment call, versus 2.23s without ONNX. A 25% improvement.
Will test it on a GPU.
Right now testing a convertion to OpenVino to see if I can usage face_alignment on CPU.
I made this notebook to assess the performance.
https://colab.research.google.com/gist/weltonrodrigo/e3e9addcd32876f7f49816eb1cd070f9/face_alignment_onnx_analysis.ipynb
