How to create 3D reconstruction from the given code

[geekysethi]

I run the th testNet_pano_full.lua in my system which is running perfectly, but I just wanted to know which script should I run to create 3D reconstruction.

[zouchuhang]

To create 3D reconstruction you need to go to "matlab" folder and run "panoOptimization.m". This shows our final 3D layout reconstruction result.

[geekysethi]

@zouchuhang thank you so much for replying, I run the panoOptimization.m and got this output but this, not the 3D reconstruction can you please help me.

[zouchuhang]

@geekysethi We do not have official 3D visualization code, the matlab file panoOptimization.m provides the 3D vertices for the layout, you can create a layout mesh with the 3D vertices and view that with any viewer.

[geekysethi]

can you please suggest some software that I can use that would be very helpful.

[alexcolburn]

Meshlab is a very useful tool for viewing and manipulating polygon meshes.

[geekysethi]

Thank you so much for replying, I will definitely try out using Meshlab.

[chaoer]

@zouchuhang Hi, I run the panoOptimization.m and find many variables after each iteration.

As you said, this script provide the 3D vertices for the layout.

But, I was wondering whether which variable stored the 3D vertices? the variable xyz ? If its true, whether variable uv was the one stored the coordinate of texture ?

@geekysethi had you already write a script for 3D visualization ?

[zouchuhang]

@chaoer Variable "cor_id" stores the 2D projection on the RGB panorama of the 3D vertices. If you want to get the 3D vertices, see "trans_eval.m", the scale of the 3D layout shape is proportional to the camera height "c_h" variable.

[zoubochao]

can I know what is the viewer you used in your Youtube video? "Visualization of (CVPR'18) LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image" @zouchuhang

[alexcolburn]

@zoubochao

The viewer is fairly simple to reproduce. Just render the room geometry and use the spherical panorama as a texture. It is really just a slightly modified panorama viewer.

I'll try to release a cross platform version when I get time.

[zoubochao]

@alexcolburn Hi Alex, I still stuck in the reconstruction step, can you share part of code that you already have,(dosen't have to be cross platform) so I can get a hint. really appreciate it. and the second question is: if i understand correctly, the output of deep net should be a 16 vector (sw, sl, sh, tx, tz, rθ) , but the mat file(e.g. 1.mat, 2.mat...) in result/box/ is 26 matrix, how should I interpret those matrices?

[lyzband]

@zouchuhang @alexcolburn @zoubochao

hello there, and same question here what should I do after I run panoOptimization.m ? or how I run meshlab?

[zouchuhang]

@zoubochao The outputs of the deep net are the two maps, the layout boundary map and the layout corner map. We then fit a 3D cuboid (see panoOptimization.m ) that best fit the two maps when projected back to 3D.

[zouchuhang]

@lyzband Please check https://github.com/zouchuhang/LayoutNet/issues/3#issuecomment-447431677 and https://github.com/zouchuhang/LayoutNet/issues/3#issuecomment-399152402. For meshlab tutorial you can refer to: http://www.meshlab.net/

[sunset1995]

I have implemented a pure python script to visualize the 3D layout using point cloud. You just need to give it the RGB panorama and the extracted cor_id store in txt. Here is the code

Demo: