Optimizing camera undistortion

[DreekFire]

Only runs Newton's method until sub-pixel accuracy is reached instead of all 10 iterations. Then resamples images with bilinear sampling to get pixel color at points with error.

Also uses analytical formulas for pixel areas instead of distorting coordinates of neighboring pixels to get pixel area.

[DreekFire]

Need example of datasets with different camera types to finish testing. Also contains lots of junk from merging branches, will clean up.

[liruilong940607]

Thanks @DreekFire for the updates. This is not an easy one.

My main question is on the fisheye formula. Have you run any tests to verify the fisheye formula is correct?

The reason I'm asking this is that the old version actually have a correct fisheye undistortion formula by combining https://github.com/nerfstudio-project/nerfstudio/blob/43c399e71ead6a9a733e29efa1c5966da3ff4629/nerfstudio/cameras/camera_utils.py#L411

and

https://github.com/nerfstudio-project/nerfstudio/blob/43c399e71ead6a9a733e29efa1c5966da3ff4629/nerfstudio/cameras/cameras.py#L724-L735

And this PR seems to change the radial_and_tangential_undistort function but not the second part for fisheye. I'm not sure if the formula is correct anymore.

[kerrj]

What's the speed/quality difference on this PR?

[DreekFire]

Some tests for correctness:

• Visually inspected results on floating-tree and poster datasets - looks good (todo: equirectangular and stereo, even though these have no undistortion, just to make sure nothing broke)
• Examined max, mean, and min pixel_areas under new method and old method: approximately matches (expect some difference because new method uses more accurate analytical formula which accounts for parallelogram-shaped pixels)
• Bilinear sampling: manual inspection on dummy data
• Measured max difference between rays returned by new and old methods: within specified tolerance for perspective camera, different for fisheye camera BUT new method is more accurate for fisheye camera (from ~1e-3 mean abs error to ~1e-8 (less than floating point precision) mean abs error.

And speed: Speed fluctuates, measured manually at two different iteration ranges: Units in thousand rays per second, on puppy.bair Iteration 500-1800: 133-137 new fisheye, 131-132 new perspective, 126-127 old fisheye, 125-127 old perspective Iteration 2000-3000: 150 new fisheye, 143-145 new perspective, 135-140 old fisheye, 133-135 old perspective Speedup not as great as before. During development, I merged a new version of nerfstudio. Before and after the merge, the new method ran at about the same speed, but the older version sped up a bit, so the difference is not as great as expected.

[kerrj]

what machine are you testing on? is there any particular sort of hardware this implementation is optimized for? eg would you see stronger benefits on weaker CPUs or GPUs, etc

[tancik]

@kerrj Will this conflict with any of the datamanager stuff that you are working on?

[DreekFire]

Ended up removing the resampling because it didn't appear to be worth it - once a ray was already within sub-pixel accuracy, Newton's method usually achieved very good accuracy in just one more iteration. This makes the code simpler as well. Now, the key optimizations are early stopping for undistortion and analytical formulas for pixel areas instead of numerical ones (should also be more accurate because it correctly accounts for skewed pixels).

The speed fluctuates throughout training, but it is generally about 10% faster at any given iteration than the main branch on puppy.

Not sure why the tests are failing - seems to be something going wrong with torch.compile. It passes on both my local machine as well as my env on the puppy machine.

[anc2001]

Thanks @DreekFire for the updates. This is not an easy one.

My main question is on the fisheye formula. Have you run any tests to verify the fisheye formula is correct?

The reason I'm asking this is that the old version actually have a correct fisheye undistortion formula by combining

https://github.com/nerfstudio-project/nerfstudio/blob/43c399e71ead6a9a733e29efa1c5966da3ff4629/nerfstudio/cameras/camera_utils.py#L411

and

https://github.com/nerfstudio-project/nerfstudio/blob/43c399e71ead6a9a733e29efa1c5966da3ff4629/nerfstudio/cameras/cameras.py#L724-L735

And this PR seems to change the radial_and_tangential_undistort function but not the second part for fisheye. I'm not sure if the formula is correct anymore.

Hi @liruilong940607, would you mind elaborating on why the old version has the correct fisheye undistortion. I'm a little confused as to how the math is correct in this case. The radial_and_tangential_undistort function implements undistortion different from the OpenCV fisheye distortion specification and it's unclear to me how the second part corrects this.

[DreekFire]

Hi @anc2001, nerfstudio does not use all of the distortion parameters simultaneously. For perspective cameras, it uses two radial and two tangential distortion parameters, while for fisheye cameras, it uses 4 radial distortion parameters. The old formula produces results identical to the formulas implemented in OpenCV (OpenCV uses different formulas for perspective and fisheye cameras) as long as you do not try to use the third or fourth radial distortion parameters for a perspective camera, or any tangential distortion parameters for a fisheye camera.

[anc2001]

Hi @DreekFire thanks for the reply. At the time of my original comment I was unsure that the current implementation of fisheye undistortion matched up with the math described in the OpenCV documentation. I worked it out and understand why it's correct now. Are you referring to the implementation on this branch or the current implementation on the main branch? To my understanding the current implementation uses 4 radial distortion parameters and 2 tangential where the 2 tangential are just 0 when the camera is fisheye. For perspective undistortion under the OpenCV distortion specification it will be incorrect if the 4th radial distortion parameter is included, but shouldn't it be correct up to the 3rd radial distortion parameter? Also shouldn't nerfstudio support the full OpenCV perspective camera model (maybe minus the thin prism coefficients)?