Not an issue, question about fisheye distortion.

[KamiCalcium]

Hi @ethanweber ,

What was the formula you used to implement the fisheye ray direction model here? I've only seen theta = atan(r), where r = sqrt(x^2+y^2) model for fisheye before and cannot understand your code here. I will appreciate it if you explain it a bit or share pointers for the derivation. Thank you very much! https://github.com/nerfstudio-project/nerfstudio/blob/3b8e2ce9ec4750b21b6b673bf62bedb61e1da2cf/nerfstudio/cameras/cameras.py#L658-L669

[KamiCalcium]

Related to https://github.com/nerfstudio-project/nerfstudio/issues/868#issuecomment-1396326938 as well.

Hi @jake-austin, I make a new issue here.

[qhdqhd]

same question... Did you solve it?

[KamiCalcium]

same question... Did you solve it?

My two cents:

The terminology distort and undistort are somewhat confusing here. In opencv's documentation: There are two steps for fisheye distortion: i) radial and tangential distortion, ii) applying fisheye effect. Let us not consider radial and tangential distortion for now, i.e., only consider fisheye effect. That said, all distortion coefficients k are 0, thus x'=(θ/r)a, and y'=(θ/r)b, where θ=atan(r), means we use the atan model for fisheye effect. Note that here the mapping from projected 2D coordinates (a, b) to distorted 2D projection coordinates (x', y') is called distort. Now, we do a simply transform: tan(θ)=r, we then have a=x'tan(θ)/θ and b=y'tan(θ)/θ. This is called undistort: mapping (x', y') -> (a, b).

Now, in nerf training process, given a fisheye image (already distorted), we want to find the ray coordinates correspondence so that we need to do undistort. Check here: https://github.com/nerfstudio-project/nerfstudio/blob/3b8e2ce9ec4750b21b6b673bf62bedb61e1da2cf/nerfstudio/cameras/cameras.py#L667-L669, if everything divided by cos(theta), it becomes:

 directions_stack[..., 0][mask] = torch.masked_select(coord_stack[..., 0] * tan_theta / theta, mask).float() 
 directions_stack[..., 1][mask] = torch.masked_select(coord_stack[..., 1] * tan_theta / theta, mask).float() 
 directions_stack[..., 2][mask] = -torch.masked_select(1, mask).float() 

, which is the same as our previous undistort derivation :)

However, there still remain confusions:

1. Why do we define theta as https://github.com/nerfstudio-project/nerfstudio/blob/3b8e2ce9ec4750b21b6b673bf62bedb61e1da2cf/nerfstudio/cameras/cameras.py#L662, shouldn't it be r instead of theta?
2. As I mentioned above, why do we do that multiplying by cos(theta) operation to make it sin(theta) instead of directly using tan(theta)?

It would be really helpful if @ethanweber and @jake-austin can correct me and explain some more details :D

[atonderski]

I am also wondering about this