Code correspondence for relative transfer in the appendix

[kenmbkr]

I was looking at relative transfer in the appendix but I have difficulty following the corresponding code.

if use_relative_movement:
    kp_value_diff = (kp_driving['value'] - kp_driving_initial['value'])
    kp_value_diff *= adapt_movement_scale
    kp_new['value'] = kp_value_diff + kp_source['value']

    if use_relative_jacobian:
        jacobian_diff = torch.matmul(kp_driving['jacobian'], torch.inverse(kp_driving_initial['jacobian']))
        kp_new['jacobian'] = torch.matmul(jacobian_diff, kp_source['jacobian'])

If I am not wrong, this line is equation 29.

jacobian_diff = torch.matmul(kp_driving['jacobian'], torch.inverse(kp_driving_initial['jacobian']))

This line should be equation 28

kp_new['jacobian'] = torch.matmul(jacobian_diff, kp_source['jacobian'])

I don't get why kp_source['jacobian'] corresponds to the four terms in the latter part of equation 28.

How do these three lines map to the rest of the equations?

kp_value_diff = (kp_driving['value'] - kp_driving_initial['value'])
kp_value_diff *= adapt_movement_scale
kp_new['value'] = kp_value_diff + kp_source['value']

I would appreciate it if you can kindly explain the missing parts.

[AliaksandrSiarohin]

"This line should be equation 28"

No this line is equation one line above without a number.

"How do these three lines map to the rest of the equations?"

Disregard for a moment adapt_movement_scale. Then these values is just 3 terms that get multiplied with J_k.

Adapt movement scale is heuristic for faces of different sizes.

[kenmbkr]

Then these values is just 3 terms that get multiplied with J_k.

So J_k is equation 29 (jacobian_diff), and kp_source['jacobian'] are the 3 terms according to you. However, kp_source['jacobian'] is not updated in the code and used as it is. I still don't get how kp_source['jacobian'] maps to all three terms in equation 28.

How do the following terms for computing relative value correspond to the first two equations $\tau _{S_t \leftarrow R} (p)$ and $\tau _{S_t \leftarrow R} (p_k)$ in the picture?

• kp_driving['value']
• kp_driving_initial['value']
• kp_source['value']

EDIT: It seems to me that this line corresponds to the three terms in equation 28, but it is not multiplied by $J_k$ in the code.

kp_new['value'] = kp_value_diff + kp_source['value']

[AliaksandrSiarohin]

You should note that my interface of "dense motion network" expects 2 sets of points kp_source and kp_driving. Later inside dense motion kp_source['jacobian'] is multiplied with kp_driving['jacobian'], the resulting J_k will be as in Eq (29).

kp_driving['value'] - T_{Dt<-R}(p_k). kp_driving_initial['value'] - T_{D1<-R}(p_k) kp_source['value'] - T_{S<-R}(p_k)

It multiplied by J_k later in dense motion network.

See https://github.com/AliaksandrSiarohin/first-order-model/blob/ca49071ce60051200f58f03a0e4e65e675a54e27/modules/dense_motion.py#L59

[kenmbkr]

You should note that my interface of "dense motion network" expects 2 sets of points kp_source and kp_driving. Later inside dense motion kp_source['jacobian'] is multiplied with kp_driving['jacobian'], the resulting J_k will be as in Eq (29).

kp_driving['value'] - T_{Dt<-R}(p_k). kp_driving_initial['value'] - T_{D1<-R}(p_k) kp_source['value'] - T_{S<-R}(p_k)

It multiplied by J_k later in dense motion network.

See

https://github.com/AliaksandrSiarohin/first-order-model/blob/ca49071ce60051200f58f03a0e4e65e675a54e27/modules/dense_motion.py#L56

I checked the code dense motion network as you suggested. I think the implementation corresponds to only equation 21, which handles only one set of driving keypoint $T_{D_t<-R}(p_k)$ (only the current frame but not the first frame). Equation 28 indeed has two sets of driving keypoints, kp_driving and kp_driving_initial, which does not match the code of dense motion.

What about these two lines? How do they correspond to $\tau _{S_t \leftarrow R} (p)$ and $\tau _{S_t \leftarrow R} (p_k)$ in the picture?

kp_value_diff = (kp_driving['value'] - kp_driving_initial['value'])
kp_new['value'] = kp_value_diff + kp_source['value']

[AliaksandrSiarohin]

When you run dense motion in relative mode, kp_new will be used instead of kp_driving. In dense motion indeed Eq (21) is implemented. You can verify that if you put in Eq (21) values of kp_new instead of kp_driving, you will get Eq (28).

[kenmbkr]

When you run dense motion in relative mode, kp_new will be used instead of kp_driving. In dense motion indeed Eq (21) is implemented. You can verify that if you put in Eq (21) values of kp_new instead of kp_driving, you will get Eq (28).

Okay. So the three terms in equation 28 is basically kp_new plugging into equation 21: replacing kp_drving by kp_new.

How do these lines map to the equations in the appendix?

kp_value_diff = (kp_driving['value'] - kp_driving_initial['value'])
kp_new['value'] = kp_value_diff + kp_source['value']

From my understanding of the code it computes the relative translation between two driving frames and applies it to the source. However, I couldn't get how it corresponds to the equations in the appendix.

Sorry that I have asked a few times above but couldn't get an answer for this part. I will really appreciate it if you can kindly clarify it.

[AliaksandrSiarohin]

As I told already:

kp_driving['value'] - T_{Dt<-R}(p_k). kp_driving_initial['value'] - T_{D1<-R}(p_k) kp_source['value'] - T_{S<-R}(p_k)

[kenmbkr]

As I told already:

kp_driving['value'] - T_{Dt<-R}(p_k). kp_driving_initial['value'] - T_{D1<-R}(p_k) kp_source['value'] - T_{S<-R}(p_k)

Let me rephrase my question. First, there are multiplications in the equations, but no multiplications in these two lines in the code. Where are the missing multiplications in the code?

kp_value_diff = (kp_driving['value'] - kp_driving_initial['value'])
kp_new['value'] = kp_value_diff + kp_source['value']

What are the corresponding parts in the code for the following terms in the equations:

• $\tau_{D_t \leftarrow D_1}$
• $\tau_{S_1 \leftarrow R} (p)$ (Note the $p$, not $p_k$ as you told me)
• $V_{S_1 \leftarrow D_1}(p_k)$

Why is kp_driving['value'] $\tau_{D_t<-R}(p_k)$ missing in the above two equations?

[AliaksandrSiarohin]

Multiplications is done inside dense motion. You looking for one to one correspondence between code and paper, but there is none. Code and paper are equivalent. As I told you before code assume that in dense motion you use kp_source and kp_driving, the paper however try to simplify the formulas and squash them together.

[kenmbkr]

Multiplications is done inside dense motion. You looking for one to one correspondence between code and paper, but there is none. Code and paper are equivalent. As I told you before code assume that in dense motion you use kp_source and kp_driving, the paper however try to simplify the formulas and squash them together.

From the author's perspective, I totally understand that the code and the paper are faithfully equivalence of each other. However, when readers come across those two equations in the appendix, it is not trivial for one to conclude that it is just those two lines in the code and the rest is packed in dense motion. A little bit of explanation on the simplification process definitely helps. I hope you don't mind.

[AliaksandrSiarohin]

I understand, but don't know how to explain it better.