Alternative keypoints

[fpandza]

Hello,

I have a few questions regarding the project. I'm struggling to find how model finds the keypoint values. I found with some code tinkering that you get 10 keypoints as per config that you then pass to generator along with jacobian values of those 10 keypoints if I'm not mistaken.

Do you have pretrained model for keypoint extraction or is that learned behavior in training? Also, if I wanted to plug in my own 'keypoints' obtained from pretrained alternative keypoint extractor and train on those, would something like that be easily doable? And how long did training( (for 100 epochs?) take you on 4 GPUs approximately? Finally, a bit unrelated, using your video preprocessing steps, I prepared vox celeb mp4s and even tried a few videos in png format, but cannot find anywhere the pairs.csv mentioned in config file and in code, how do you obtain that file and is it necessary?

Thank you!

[AliaksandrSiarohin]

Keypoint detector is unsupervised, and learned end to end with an entire system. You can replace keypoint detector with pretrained one if you want. Pairs.csv is not needed. I used them to specify files for evaluation.

[fpandza]

Thank you for a quick reply!

Could you also share how long did training for 100 epochs last on VoxCeleb dataset approximately and what GPU config did you use? Also, did you use png or mp4 format?

[mdv3101]

@fpandza Did you able to train the model with custom keypoints detected from a separate model?

@AliaksandrSiarohin I want to replace the keypoints with some third-party model, like dlib. Can you suggest the necessary steps?

Thanks

[AliaksandrSiarohin]

Do you plan to estimate keypoints during training or prior to it?

[mdv3101]

I am planning to use dlib for estimating the keypoints and use them to train a motion model. For every image, instead of training a unsupervised model for getting keypoints, I am thinking to use dlib.

[mdv3101]

Hi @AliaksandrSiarohin, In keypoint_detector.py, I made some changes to create a key-point map of [1,68,58,58] where each 58x58 matrix has all zeros except one. The location where dlib keypoint is detected, is set to 1 in the above tensor. Now when I try to train it, I got the error:

Traceback (most recent call last):
  File "run.py", line 81, in <module>
    train(config, generator, discriminator, kp_detector, opt.checkpoint, log_dir, dataset, opt.device_ids)
  File "/home/madhav3101/gan_codes/first-order-model-dlib/train.py", line 52, in train
    losses_generator, generated = generator_full(x)
  File "/home/madhav3101/env_gan/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/madhav3101/env_gan/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 159, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/madhav3101/env_gan/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/madhav3101/gan_codes/first-order-model-dlib/modules/model.py", line 210, in forward
    jacobian_transformed = torch.matmul(transform.jacobian(transformed_kp['value']),
  File "/home/madhav3101/gan_codes/first-order-model-dlib/modules/model.py", line 117, in jacobian
    grad_x = grad(new_coordinates[..., 0].sum(), coordinates, create_graph=True)
  File "/home/madhav3101/env_gan/lib/python3.7/site-packages/torch/autograd/__init__.py", line 204, in grad
    inputs, allow_unused)
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

I guess, the error is occurring because keypoints detected by dlib are static and don't have anything to backpropogate. The jacobian is trying to get the gradient of it. Any suggestion on how to solve this or any other approach to use dlib keypoints?

[AliaksandrSiarohin]

Set equivariance loss to zero.

[mdv3101]

Hi @AliaksandrSiarohin , Thanks for your suggestion. I set equivariance_value: 0 and equivariance_jacobian: 0. The code starts working. The perceptual loss also starts reducing. I checked the visualization and it shows some strange output (image below). Can you suggest, what has gone wrong? Is it because of the horizontal_flip parameter or something else?

[AliaksandrSiarohin]

It is because x and y axis is inverted. You should invert your keypoints axis, e.g. kp = kp[..., [1,0]]. Also you should not use horizontal flip, unless you also flipping the keypoint.

[wjtan99]

@mdv3101 Can you share details or code how you implement it? Thanks a lot.

[ghost]

Hi @mdv3101, @fpandza, did you manage to get good results after replacing the unsupervised keypoint detector with a custom KP detector ? Thanks a lot !! :)

[RAJA-PARIKSHAT]

Keypoint detector is unsupervised, and learned end to end with an entire system. You can replace keypoint detector with pretrained one if you want. Pairs.csv is not needed. I used them to specify files for evaluation.

@AliaksandrSiarohin Can you explain how can I replace keypoint detecter with my own pretrained detector and use it with the remaining model. Basically I couldn't figure out how to estimate jacobian in that case.

[AliaksandrSiarohin]

You may not use jacobians.

[RAJA-PARIKSHAT]

@AliaksandrSiarohin wouldn't that decrease performance, if I don't estimate jacobians, because these jacobians are also used in the subsequent code as well

[AliaksandrSiarohin]

Supervised keypoints will decrease it anyway.