How to FaceFormer output arkit blendshape coefficient instead of vertices?

[youngstu]

The work is remarkable. How to FaceFormer output arkit blendshape coefficient instead of vertices? Very Thanks.

[youngstu]

If faceformer supports arkit-blendshape-coefficient output, it can support other 3D templates.

[lucasjinreal]

@youngstu Did u get a way to do this? I got some problem

[EvelynFan]

The work is remarkable. How to FaceFormer output arkit blendshape coefficient instead of vertices? Very Thanks.

Hi, for the current model, the output representation is vertex positions. The training datasets we used do not contain the blendshape coefficients ground truth, so the model was not able to learn such information during training. I guess some other techniques based on inverse graphics may be required to convert the vertex positions into 51 blendshape coefficients.

[zhaiyuan0217]

The work is remarkable. How to FaceFormer output arkit blendshape coefficient instead of vertices? Very Thanks.

Hi, for the current model, the output representation is vertex positions. The training datasets we used do not contain the blendshape coefficients ground truth, so the model was not able to learn such information during training. I guess some other techniques based on inverse graphics may be required to convert the vertex positions into 51 blendshape coefficients.

I want to know if i have the blendshape coefficients groud truth, why in the render video, the lip animation is not evident

[zhaiyuan0217]

@youngstu Did u get a way to do this? I got some problem

what's problem did you encounter, and did you solve it

[lucasjinreal]

@zhaiyuan0217 not yet, did u able to get 51 blendshapes ?

[zhaiyuan0217]

@zhaiyuan0217 not yet, did u able to get 51 blendshapes ?

we have blendshapes with our data, after traing the model with our data and blendshapes, the rendered video is not well, but traing the model with vertices, the rendered video is well.

[lucasjinreal]

@zhaiyuan0217 Hi, what do u mean training the model with vertices

[zhaiyuan0217]

i mean the output is 5023*3 vertices.

[xiaodongyichuan]

@zhaiyuan0217 not yet, did u able to get 51 blendshapes ?

we have blendshapes with our data, after traing the model with our data and blendshapes, the rendered video is not well, but traing the model with vertices, the rendered video is well.

i have same question

[Shirley-0708]

May I ask how you create your own dataset? Can you be specific? Thank you！ @zhaiyuan0217 @xiaodongyichuan

[zhiyoujingtian]

@zhaiyuan0217 not yet, did u able to get 51 blendshapes ?

we have blendshapes with our data, after traing the model with our data and blendshapes, the rendered video is not well, but traing the model with vertices, the rendered video is well.

Did you solve this problem? I also trained with wav and blendshape data, and the final driving effect is that the mouth basically does not open

[agupta54]

I was trying to train on DECA expressions but the mouth is not moving when rendering the result.