Using SceneGraphFusion with custom dataset

[MihirDharmadhikari]

Hello,

Thank you very much for this work! I wanted to use this package on a custom dataset. I don't plan to retrain the network but use the pre-trained network provided with the code. In the current structure, the package requires the data in the3RScan or ScanNet format with the segmented mesh of the environment. I wanted to ask what would be the best way to use the package without having the segmented mesh.

Best, Mihir

[ShunChengWu]

Hi Mihir,

the backend runs with a given set of nodes and their neighbor graph for the semantic scene graph estimation. You can ignore the front-end part, which is the incremental segmentation, and use only the backend by creating the nodes to update and their neighbor graph.

Best, Shun-Cheng

[MihirDharmadhikari]

Hi Shun-Cheng,

In the code, I see that while using the 3RScan dataset, the method creates the depth image by raycasting into the mesh. However, when I disable that and use the depth image provided in the dataset I don't see any graph being built. Here is a snapshot of the GUI when the depth image from mesh rendering is disabled:

Here is when it uses the depth image from the mesh renderer:

I am disabling the depth image generation from the mesh by commenting out this line (when not using the GUI, and I comment out the similar line in the GUI code when using it). Is there a way to use the depth images in the dataset directly? Also, do let me know if the way I am disabling the depth image generation from the mesh is correct.

Best, Mihir

[ShunChengWu]

Hi Mihir,

The original depth from the 3RScan dataset is terrible. That's why I had to use rendered depth from the mesh. The way you disabled the depth is okay. A better way to do is to disable renderer here https://github.com/ShunChengWu/SceneGraphFusion/blob/5bf9017c00949aedca1430854240667b3fa06565/app/GraphSLAM/main.cc#L219-L225 The purpose of this renderer is to render a depth map from a given mesh. If you intend to use the raw depth from your dataset, then you don't need this renderer at all. Just write a dataloader (by reading the RGB, Depth and pose to m_rgb, m_d and m_pose respectively) and put your loader into the factory (https://github.com/ShunChengWu/SceneGraphFusion/blob/5bf9017c00949aedca1430854240667b3fa06565/libDataLoader/include/dataLoader/dataset_loader_facotry.h).

[MihirDharmadhikari]

Hi Shun-Cheng,

Thanks for this info! I will try this out and get back to you.

Best, Mihir