gsplat icon indicating copy to clipboard operation
gsplat copied to clipboard

Add gradients to intrinsics

Open nlml opened this issue 5 months ago • 0 comments

This PR modifies FullyFusedProjection and FullyFusedProjection2DGS such that the intrinsics (Ks) receive gradients. Am currently working on adding it to PackedProjection as well

Inspiration and derivation of gradients to K for 3DGS came from this paper.

2DGS gradients were derived by hand.

A simple test is added for now. It confirms that an "incorrect" K converges to the correct K from RGB gradients alone.

python test_intrinsics_optimization.py --use_3dgs
python test_intrinsics_optimization.py --use_2dgs

The results will look something like this: image

However, while I am fairly confident, I am not 100% sure on the math of the gradients. Would be great if someone can double-check!

Not too sure on the design choice of overloading the persp_proj_vjp function for 3DGS to support with/without Ks grads. Maybe we can change that.

nlml avatar Sep 23 '25 02:09 nlml