gradient/ laplacian supervision

[etienne87]

hello, thanks for this implementation. how would you go about implementing the gradient or laplacian supervision? i wonder if it is involving the gradient of the network directly or some sort of sobel filtering over outputs (sample patches instead of pixels?), which would be much less elegant i guess?

[scart97]

I have not tried to implement this supervision, so I'm not sure how it will happen. According to the paper, section 3.2 of the appendix:

The ground truth gradient image is computed using the Sobel filter, and is scaled by a constant factor of 10 for training. The ground truth Laplace image is computed using a Laplace filter, and is scaled by a constant factor of 10,000 for training.

And also:

We train for 10,000 iterations, and at each iteration fit on every pixel in the gradient or Laplacian image.

But I could not find how they calculate the gradient of the Siren output. It's worthy taking a look at other implementations, and see if anyone got this working. If you find anything please link it here.

[etienne87]

i was thinking to use torch.autograd.grad with only_inputs=True, but i never used this feature and it seems you need to call it N times otherwise RuntimeError: grad can be implicitly created only for scalar outputs. there is also a "jacobian" function, will keep you informed if i find how to do it.

[etienne87]

i did a first version just with numerical gradient (not the complicated autodiff stuff yet) https://gist.github.com/etienne87/0519972704e711c20706f56d55b63890

grad = [None,None]
for r in [-1,1]:
  for dim in [0,1]:
      bx2 = bx.clone()
      bx2[:,dim] += r/100
      o = net(bx2)
      if grad[dim] is None:
         grad[dim] = torch.zeros_like(o)
      grad[dim] += r * o

it more or less learns the image (when you train without the rgb supervision) up to a constant. it does not really seem to help when combined.

with just rgb supervision it is better so far:

[scart97]

That's cool!

I also tried to get this working today, extracting patches from the image and using sobel filtering. My model learned the correct gradients and also started to get decent results on the laplacian, but the RGB colors are totally wrong.

Gradient of predictions:

RGB predictions:

[etienne87]

~~but you learn directly the gradient, or make so that the gradient of your network match the gradient of the ground truth?~~

ok so you also learn with a "numerical gradient right? (output prediction for all pixels and run sobel filtering on it before comparing it to ground truth)

[scart97]

Yeah, I compare the numerical gradient of the network output against the ground truth.

The official implementation is out, and they use autograd to calculate the networks gradients.