first-order-model
first-order-model copied to clipboard
AliaksandrSiarohin
A few basic question on implementation detail.
Thanks for your work ! Here are some questions:
-
About formula 4 and your implementation.
Is this right for each kind of color of variabel match with the original formula ?
In my undestanding, the references frameRmeans the origin point ofKdim space in virtual coordinates, andzdenotes "the space coordinate of D", but in your code it's a standard meshgrid, theidentity_grid. -
About
HourGlassI don't see theHourGlassmodule in your paper, but in your code https://github.com/AliaksandrSiarohin/first-order-model/blob/902f83a4217c75c842e1f536b3331c5032b703a2/modules/dense_motion.py#L15
What does it stand for ? Cause we alrerady have a) the warp module b) the occlusion module.
I will appreciate for your answer ~
- Colors are right.
- I need to compute the source coordinate for each driving coordinate (z). So identity grid is just contain all the driving coordinates in a grid, e.g [-1,1]x[-1,1].
- Hourglass is just a type of architecture. In paper it is called Unet.
Thanks for quick response, I will check the code part of z.
After carefully read your paper, I still cannot understand why use the reference frame R. What's the benefit of using S<-R<-D rather than computing directly S<-D ? Sorry for the stupid question.
The lack of paired data make this approach not meaningful. So imagine you train a network to predict a transformation S<-D from [S ¦¦ D], S concatenated with D. At training time you can only use frames from the same video, while at test time you will need to use frames from different videos. The network will likely never generalize to frames from different videos, since it never saw any.
To this end we trying to learn a network that make an independent predictions for S and D. And to define it properly we introduce this R.
If the purpose of introducing R is to prevent "train on same video, test on different", can we just use random [S, D] pair which from different video to train network, without using R ?
Also, currently your model still train D and S from the same video( if I'm not wrong), therefore how to prevent the situation you mentioned ? https://github.com/AliaksandrSiarohin/first-order-model/blob/902f83a4217c75c842e1f536b3331c5032b703a2/frames_dataset.py#L110-L114
No this is not the purpose. The purpose is to make independent motion predictions for S, D. If the motion predictions is dependent on each other, e.g. Keypoint predictor use a concatenation of S and D, it won't generalise.
"I need to compute the source coordinate for each driving coordinate (z). So identity grid is just contain all the driving coordinates in a grid, e.g [-1,1]x[-1,1]."
Here I still can't understand the difference between driving coordinate (z) and kp_driving. I think kp_driving means the relative coordinate of K keypoints from R->D, and the driving coordinate means the local pixels around each z(k).
If so, why we can reprersent z with an identity_grid ?
Yes true z is local pixel around z_k. However at the point where I produce this sparse motions, I don't know what will be the neiboirhoods. So I compute the transformation for all the coordinates in driving, and select the neiboirhoods afterwards. Note that all possible coordinates in the driving frame that we potentially may need for warping can be produced by identity_grid.
And, if the purpose is tto make independent motion predictions for S, D. Then I think we should optimize D and S independently, such as we optimize (R, S) and (R, D).
But in your code, the R is always used as an intermediate variable for S<-D. we do not add restrict to R, so how can we make sure D and S trained independently ?
And, if the purpose is tto make independent motion predictions for S, D. Then I think we should optimize D and S independently, such as we optimize (R, S) and (R, D).
But in your code, the R is always used as an intermediate variable for S<-D. we do not add restrict to R, so how can we make sure D and S trained independently ?
Later only motion information from driving is used. So prediction will be independed of the driving appearance.
Guess it will be easier if you say how you want to implement it, and I will say why this would not work.
OK, for example:
1.Select random frame D0 and S0 which are similar from different videos. 2.For each frame of Di videos, compute keypoints and heatmap, and the sparse motion between(D0, Di). 3.Predict new S from (S0, spare motion) with Di motion and S0 appearance.
Currently I havn't totally understand your paper, so maybe I've gone to a wrong direction.
Yes, this would be ideal training scheme. Note that for step 3 you need a ground truth for S. So the training data required is videos (S and D) where 2 object perform exactly the same movements, and it is not possible to find it from in the wild videos.
But why I need full S video? I only use S0 in the whole training phase.
You mean the ground truth S for reconstruction loss, which is the same appearance with S0 and same motion with Di ?
But why I need full S video? I only use S0 in the whole training phase.
The network is trained with reconstruction loss, so you will need to compute
|| S - \hat(S)||
Where \hat(S) is ground truth video.
Now I understand.
But What if I can warp S0 with sparse motion between D0 and Di in step 2 ? So the warped new S0 can be the ground truth.
Its the similar idea with yours, but without the use of R.
In that case what will be your training signal for obtaining sparse motions between D0 and Di, in the first place?
I think the sparse motion can be direrctly computed from (kp_Di, kp_D0, D0) ?
Yes, but the whole purpose of the paper is to avoid it. To be able to train on arbitrary objects.
Though until now I didn't recognized that keypoint training is associated with motion module. Before now I thought keypoint training is an independent part.
So here is the purpose why use R: because we do not have a labeled keypoint dataset, So we need to "assume" there exist a standard frame R for each pair image (D, S), which is the origin point of a K-dim space . Then we can train Keypoint Detector and Motion Module in an unsupervised method.
Currently I have a good labeled keypoint dataset and pretrained model, can I replace Keypoint Detector with this pretrained model ? What's the advantage of a learned keypoint in an unsupervised way ?
-
More or less. Not sure why it is K-dim, it more like K two-dimensional coordinate systems.
-
Sure you can replace. Most of the times supervised key-points works better. Unsupervised key-points however can describe the movements that you may forget to annotate. Plus unsupervised keypoints can encode more staff per keypoint, for faces people usually use 68 supervised keypoints. While here I use 10.
Thanks a lot. I've finished reading your paper but with a lot questions. Will explore the code for more precise understanding.
Sorry for wasting so much of your time, still appreciate your kindly explanation.
I think there are some mistake in your video link , which you reverse the picture of training and testing.
This shoud be training,
and this should be testing.
Am I right ?
Hi, why you learn the jacobian rather than compute directly. is this because that the transformation from D<-R and S<-R remains unknown yet ?
https://github.com/AliaksandrSiarohin/first-order-model/blob/6e18130da4e7931bc423afc85b4ea9f985a938c2/modules/keypoint_detector.py#L25
https://github.com/AliaksandrSiarohin/first-order-model/blob/6e18130da4e7931bc423afc85b4ea9f985a938c2/modules/keypoint_detector.py#L64
No I'm not saying we "can" compute. It just a intuition that jacobian is computed rather than learned.. In this model, we don't know the exact function f(x) of the transformation from D<-R and S<-R, because your "learn" from network, so you then learn jacobian too, right ?