Does anyone have a plan to reproduce EMO?

[liuzhuang1024]

[johndpope]

progress thus far

downloaded - https://academictorrents.com/download/843b5adb0358124d388c4e9836654c246b988ff4.torrent

https://github.com/johndpope/Emote-hack

created some data loader EMODataset.py

HeadRotation.py - using mediapipe get_head_pose_velocities_at_frame

SpeedEncoder do the tanh thing to categorise head motion to speed bucket

Wav2VecFeatureExtractor using wav2vec2 (maybe it needs to be downgraded?) extract_features_from_mp4 - get nearby frames

Backbone / attention layers - WIP https://github.com/johndpope/Emote-hack/blob/main/Net.py

UPDATE (the rest of the code is scrap heap at the moment.)

the previous paper by HumanAIGC AnimateAnyone was successfully implemented by MooreThreads (includes training code) upvote here for them to recreate EMO paper. https://github.com/MooreThreads/Moore-AnimateAnyone/issues/98

this code is mostly cherry picked from magic-animate (does not include training code) - (which is based off animatediff) https://github.com/magic-research/magic-animate/tree/main

[fire17]

@liuzhuang1024 awesome was gonna ask the same

@johndpope good luck man! We're all eagerly patient to see what you come up with!

Has anyone found any other teams or people working in this? Hope to see people help each other and make this a reality. We're about to enter the golden age of open source 😁 All the best everyone!

[liutaocode]

@johndpope , great job. However, I have some suggestions after watching some videos from CelebV-HQ. You may manually filter them to ensure high resolution and remove bad cases, such as individuals wearing masks or significant background changes, etc. From my limited experience, training or fine-tuning Stable Diffusion can be quite unstable if the data contains too many bad cases.