emonet_model_path

[filby89]

I am trying to setup a training environment for EMOCA, however in emoca.yaml the "emonet_model_path" used for emotion loss is empty. How should we initialize this ?

[radekd91]

Download ResNet50 from the EmotionRecognition folder and point the path to where you download it.

Keep me posted on how it goes! :-)

On Sun, May 1, 2022, 13:48 Filntisis Panagiotis @.***> wrote:

I am trying to setup a training environment for EMOCA, however in emoca.yaml the "emonet_model_path" used for emotion loss is empty. How should we initialize this ?

— Reply to this email directly, view it on GitHub https://github.com/radekd91/emoca/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7MKHIRSACHJS6OVEESLGLVHZVPZANCNFSM5UZYRGKQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[radekd91]

@filby89 how did it go? :-)

[filby89]

Hey! In the end I actually found it hard to browse through the code (lots of options!) and decided to reimplement the crucial part of emotion loss (I have some limited success) using the resnet50 you provided. I have some questions regarding the loss:

• Do you perform any feature normalization in the outputs of emotion resnet50 before calculating their MSE ?
• Is there any preprocessing/normalization done in the rendered/input images before inputting them into resnet50 ?
• Do you add the original inner mouth/hari/background into the rendered image before extracting features ?

Thanks for your help !

[radekd91]

1. there is no image normalization apart from making sure the images are in the same range (such as [0,1])
2. nope, just as long as they're all in the same range
3. this is optional, you can inpaint the rest of the image around the rendered face. Both should work in practice.

[radekd91]

@filby89 , EMOCA 2.0 is out and among other things it adds a lip reading loss inspired by SPECTRE. Thanks for exploring that line of research. :-)

[filby89]

Hey @radekd91, the new version seems awesome! Thank you very much for your mention also :) Very interesting that MSE loss provides more stable results and less artifacts compared to cosine!

[radekd91]

@filby89 . I suspect this happens because MSE also considers the magnitude of the feature, as opposed to just the angle. I had similar experience with the emotion loss in the emoca paper. I'd get more undesirable exaggaration artifacts, probably due to cosine similarity being minimized but the magnitude difference wasn't. It's a speculation, though.