Question about 3D reconstruction from video

[HowieMa]

Hi, thanks for this fantastic work! When reconstructing the 3D face model from a video of a subject, it seems that the current demo code reconstructs the 3D face frame by frame, as in this code.
However, in practice, I find that this implementation cannot guarantee the consistency of shape code (identity code) across different frames. Thus, I'd like to know if is there any way that can maintain the shape consistency during the inference time as well, just like the shape consistency loss in the training time. Look forward to your reply. Thanks!

[emlcpfx]

Howie, I'm wondering the same thing. Did you ever make any progress with this?

[HowieMa]

Howie, I'm wondering the same thing. Did you ever make any progress with this?

Hi, I don't think the vanilla DECA can support this function. Maybe we can use DECA to get initial parameters and then apply an optimization-based method to refine the performance on a video, like this https://github.com/philgras/video-head-tracker

[emlcpfx]

They’re using DECA in this— https://imaging.cs.cmu.edu/local_deformation_fields/

I asked them if they had modified DECA at all to get better tracking, and they said, “ We finetuned the results by projecting the tracked 3d landmarks from DECA to 2d landmarks detected from a standard landmark detector.”

[Echo-jyt]

Hi, Did you find a good solution?