Add gradients to intrinsics
This PR modifies FullyFusedProjection and FullyFusedProjection2DGS such that the intrinsics (Ks) receive gradients.
Am currently working on adding it to PackedProjection as well
Inspiration and derivation of gradients to K for 3DGS came from this paper.
2DGS gradients were derived by hand.
A simple test is added for now. It confirms that an "incorrect" K converges to the correct K from RGB gradients alone.
python test_intrinsics_optimization.py --use_3dgs
python test_intrinsics_optimization.py --use_2dgs
The results will look something like this:
However, while I am fairly confident, I am not 100% sure on the math of the gradients. Would be great if someone can double-check!
Not too sure on the design choice of overloading the persp_proj_vjp function for 3DGS to support with/without Ks grads. Maybe we can change that.