Incorporating Known Camera Poses in Global Alignment Optimization

[resurgo97]

I'm seeking a way to run global alignment optimization with known camera poses. After examining the code, I've identified some challenges:

1. cam2w matrices are constructed in make_K_cam_depth, but they don't directly correspond to standard extrinsic matrices.

2. The code handles scale ambiguity by adjusting the translation vector based on focal length and scale, rather than multiplying scales to the pointmaps.

3. This approach leverages the fact that scaling up the pointmap is equivalent to: a) increasing the focal length b) shifting the camera location along the z-axis

4. As a result, when mapping 2D points into 3D space, per-frame scales don't need to be multiplied to depthmaps:

   pts3d = proj3d(invK[img], pixels, depthmaps[img][idxs] * offsets)
   

While this implementation is convenient, it makes incorporating known camera poses challenging because cam2w is not a pure extrinsic matrix and is entangled with the focal length (an intrinsic parameter).

Questions

1. Are there plans to enable the incorporation of known camera poses?
2. Is there consideration for changing the code design to facilitate easier integration of known camera poses?

I welcome any opinions or insights on this topic from other people.

Thank you!