Potential bug on camera optimizers operation

[yuantianyuan01]

I am confused about the operation on camera optimizers at https://github.com/nerfstudio-project/nerfstudio/blob/9528a3f717fe28a6c6500199b5b37d4f2c6e10ca/nerfstudio/cameras/cameras.py#L837-L838

Instead I suppose it should be c2w = pose_utils.multiply(camera_opt_to_camera, c2w) to apply the pose adjustment in camera_opt_to_camera onto the original pose c2w.

[brentyi]

To me this looks like just confusing naming, where c2w @ camera_opt_to_camera is equivalent to world_from_camera @ camera_from_camera_opt, which gets us world_from_camera_opt / camera_opt_to_world. (does this make sense?)

IMO this brings up a good point that we should flip all of these variable names though, the A to B convention is really frustrating compared to B from A when we're using left-multiplied transformation matrices. 🙂

[yuantianyuan01]

Thanks for your response! Yes I think your explanation makes sense but I'm still confused when I'm trying to add a custom pose optimizer. Since my dataset only has pose errors on z-axis (in world coordinates), I make my custom camera_opt_to_camera something like

1 0 0 0
0 1 0 0
0 0 1 \delta z

In this case, I find c2w = pose_utils.multiply(c2w, camera_opt_to_camera) would go wrong because the transformation in z-axis is applied before the rotation in c2w while c2w = pose_utils.multiply(camera_opt_to_camera, c2w) results in correct transformation. Could you figure out how to make a custom camera_opt_to_camera that only optimizes z-axis pose in world coordinates?

[hyperzy]

@yuantianyuan01 Your instinct is correct. I was confused by this operation as well. But then I realize it has to be considered as part of the procedure of view dir generation directions_stack = torch.sum(directions_stack[..., None, :] * rotation, dim=-1) which is equivalent to wolrd_view_dir = pixel_dir @ rotation.T The transpose operation thus implies the reversed order of camera_adjustment matrix and the pose matrix.

So, this is not a bug, but a different implementation for the later matrix multiplication IMHO.