rendered features from dibr_rasterization function don't have gradient to the vertices

[rozentill]

Hi there,

I recently used dib-r as a differentiable renderer to optimize 3d shapes with guidance of the rendered image. However I found that the image features returned from dibr_rasterization function don't have any gradient to the input vertices, only the soft mask has. So the mask loss would affect the input shape but not the image loss, I assume this is not what the paper proposed. Is this what would be expected or something is wrong here? If it's supposed to be like this, then how can I optimize the shape with the help of rendered features not only the mask? Thanks.

[Caenorst]

Hi @rozentill !

Thank you for your interest in Kaolin!

I don't think this is correct, you can see in the backward code, that the first kernel here: https://github.com/NVIDIAGameWorks/kaolin/blob/master/kaolin/csrc/render/dibr_back.cu#L353-L365 is not using the soft mask gradient and is still adding some gradient values to the vertices here: https://github.com/NVIDIAGameWorks/kaolin/blob/master/kaolin/csrc/render/dibr_back.cu#L189-L196

However, the gradient will only be generated from pixel gradients where a face is intersected. The gradient will be 0. if:

1. The pixel gradient is 0. or
2. The pixel is not rasterizing any face (doesn't hit any face).

In case 2), the soft mask may be useful to generate gradients on the vertices.

[Caenorst]

Closing for inactivity