NVlabs
How to find transformation matrix between camera coordinate system (input via transforms.json file) and bounding box coordinate system?
I am also interested in backprojecting from 2D to 3D... I know in colmap that the reverse operation is [R.T, R.T.dot(t)] (source:https://colmap.github.io/format.html?highlight=inverse), however I don't think it's still applicable after swapping axes, doing the normalization and etc done in colmap2nerf.py...
If you did find a solution to this, I'm very interested in the answer
I used code to get an obj file that I wanted to reproject a three-dimensional point cloud onto a two-dimensional image using transforms.json
Hi,
sorry for not updating this issue after I found a solution quite a while ago… what I found being the biggest issue was figuring out the coordinate systems that the obj was in and the coordinate system transforms.json are described, because most likely for reprojection into 2d images, we need to transform either of them. Firstly I reproduced the point cloud using equations from a book and the transformation for this was different one required than reprojecting the mesh using pyrender, so it’s best to visually debug it, I’d recommend for anyone who tries this.
That's really good news， but I used the traditional world coordinates - camera coordinates - pixel coordinates, and the result was terrible. Can you explain in detail how that was converted.
Thank you very much！
It’s some time ago I worked with this problem so I can’t remember in top of my head by I’ll post when I get home on my desktop later today
For reprojection I can see I flipped the sign in the focal length in x-direction in camera matrix. Then inverted the transformation matrices and used cv2.rodrigues to get the rotation vector as input to cv2.projectPoints - This will give you a reprojected point cloud if you input the vertices. If you use something for projecting the mesh that uses same coordinate system as openCV, then it should be the same with the mesh.
I think the result is dependent on how well colmap/SLAM estimates your projection matrices so i would recommend visualizing them for debugging in the interactive viewer in instant-ngp first if you run on your own dataset and haven't already.
I tried the transformation according to your method, but there is still a problem