Is it possible to get a mesh reconstruction of the entire scene observed throughout the entire video?

[andrewsonga]

Hello,

as the title suggests, my question is whether it is possible to get a mesh reconstruction of the entire scene observed throughout the entire video, as opposed to getting a per-frame mesh reconstruction that isn't stitched together with meshes from other frames.

I'm hoping to use a reconstructed mesh to center the camera poses such that the world origin lies at the center of the reconstructed scene mesh, as opposed to the camera center of the first frame of the video.

Thanks!

[marek-simonik]

Hello,

I do not think it would be possible to create a static mesh out of a 3D video in general because there can be moving objects in the scene together with topology changes, which (in my opinion) cannot be captured by a static mesh.

However, you can access camera pose for each frame of a LiDAR video, so that might help you adjust the origin.

Camera poses are available e.g. in the metadata.json file after you export a 3D video into the "EXR + JPG (TouchDesigner-friendly)" format. Each pose consists of quaternion and world position ([qx, qy, qz, qw, x, y, z]).

[andybak]

The "question behind the question" might have a different solution however:

I'm hoping to use a reconstructed mesh to center the camera poses such that the world origin lies at the center of the reconstructed scene mesh, as opposed to the camera center of the first frame of the video.

I suspect there's easier ways to achieve this than a full mesh reconstruction.

[siyanhu]

I think you need colmap.