Camera rotation and translation in Meshroom-2023

[yuliaar]

The photogrammetry pipeline outputs cameras rotation and translation recovered from input images - either in cameras.sfm or their inverts in _KRt.txt files. Previously I have successfully used camera transforms recovered by Meshroom-2021 for 2D rendering with pyrender and similar openGL based renderers (it only required an additional 180 degrees rotation around x applied to Meshroom rotation matrix). Reading up on changes in Meshroom-2023 it looked like this version of Meshroom would align its coordinate frame with the openGL convention, so I expected that the only difference in how camera transforms should be applied would that the rotation around x does not need to be added during rendering. However, it is not like that at all, and I cannot figure out how to apply the camera transform produced by Meshroom 2023 to the mesh produced by Meshroom 2023 to achieve the correct 2D rendering.

In this issue, unrelated to my question, someone, however, says:

I think the reference frame convention has been changed between two releases, so if you used different versions, it could explain it. Maybe @servantftechnicolor has some better insight."

This is exactly what I would like to know, how the camera transform matrices have changed so I could amend my rendering code accordingly.

The render function below correctly works with texturedMesh.obj and _KRt.txt files produced by Meshroom 2021 but failes with files produced Meshroom 2023 (also fails without additional rotation Rx).

def render(pmesh,f,cx,cy,rotmat,tvec):
    
    # rotation by 180 around X
    Rx = np.array([[1, 0, 0],
                   [0, -1, 0],
                   [0, 0, -1]])        

    # homogeneous matrix 
    real_pose = np.eye(4,dtype=float)

    real_pose[:3,:3] = Rx @ rotmat
    real_pose[:3, 3] = Rx @ tvec

    real_pose = np.linalg.inv(real_pose)

    # setup cam
    real_cam = pyrender.IntrinsicsCamera(f,f,cx,cy)
    light = pyrender.PointLight(color=np.ones(3), intensity=10.0)
    scene = pyrender.Scene()
    obj_node = scene.add(pmesh)
    cam_node = scene.add(real_cam, pose=real_pose)
    point_l= scene.add(light, pose=real_pose)

    # render
    r = pyrender.OffscreenRenderer(viewport_width=cx*2, viewport_height=cy*2)
    color, depth = r.render(scene)
    
    return color

Desktop (please complete the following and other pertinent information):

• OS: linux suse
• Python version 3.9
• Meshroom versions:
  • 2021.1.0
  • 2023.2.0