AniPortrait icon indicating copy to clipboard operation
AniPortrait copied to clipboard

How to obtain neutral mesh during audio2mesh training?

Open nuo1wang opened this issue 1 year ago • 0 comments

Thanks for open-sourcing such a wonderful work!

I notice that in Fig.1 of the technique report, the neutral mesh is used to calculated the mesh offset during audio2lmk training. It seems that you use mediapipe to obtain the mesh vertices in this step, would you please share how you obtained neutral mesh vertices in this step?

I also notice that during audio2vid inference, you add the predicted mesh offset to the non-neutral mesh vertices obtained from the reference image, will this effect the performance of lip sync?

nuo1wang avatar Jul 26 '24 06:07 nuo1wang