Question about how the input motion phases of future frames is constructed

[SuX97]

Hi,

Thanks for your work!

I am confused by the description in paper here. I get it that during training, the phase feature is pre-extracted for each frame i from i - 60 to i + 60, but while inference, how can we extract the equivalent feature? Say, if I want to generate inbetweening frame from i to i + 120(target frame), [i - 60, i] is available, but (i, i + 60] is not available, since the future cannot be seen.

How do you solve this question?

Thanks!

[paulstarke]

Hi @SuX97 ! We observed that it's sufficient for the network to cluster the motion within this two second time window. This is also one reason why the system is able to handle those long transition windows as it runs as a time series system updated in each frame. In your example, during inference the future phase vectors in (i, i + 60] are not given as a control signal but predicted by the network itself based on the history.

[SuX97]

Hi @pauzii :

Thanks for answering!

I get it that when progressing from a NON-FIRST-SECOND frame i (i - src_idx >60), the transition network output the 7 future manifolds for 1s in the future(P_(i+10)...P_{i+60}), and the manifold input for predicting from the next frame(i + 1) can be somehow sampled from P_{i+1-60}...P_{i+1+60}, 13 frames in total. That makes sense to me.

However, if the frame is an alone key-frame in a sequence, meaning there is no keyframes in this animation except the source and target, how I can get the history of the source frame? Even if there is history frames before the source frame, how can I start progressing at the 0-index? To transit from 0 to 1, I need manifolds -60 to 60 as input. For manifold[-60], simply PAE(velocity[-120:0]) will be good. But to get manfolds -59, I need velocity from -119 to 1, the velocity for 1-frame is not available -- the first transition is not even started! To get velocity[1], I need to predict transition, but to predict, I need manifold[-59], which requires velocity[1], looks like a deadlock to me.

I have read the Input and Output format parts many times, still cannot think it straight. Hope you can help me out!

Thanks!