autonomousvision
OOM issue when i try extract_mesh_tsdf
When I try python extract_mesh_tsdf.py -m exp_playroom/release --iteration 30000, i found issues below:
Looking for config file in exp_playroom/release/cfg_args
Config file found: exp_playroom/release/cfg_args
Rendering exp_playroom/release
Loading trained model at iteration 30000
Reading camera 225/225
Loading Training Cameras
Loading Test Cameras
Rendering progress: 2%|█▍ | 5/225 [00:00<00:29, 7.35it/s]
Traceback (most recent call last):
File "extract_mesh_tsdf.py", line 107, in <module>
extract_mesh(model.extract(args), args.iteration, pipeline.extract(args))
File "extract_mesh_tsdf.py", line 90, in extract_mesh
tsdf_fusion(dataset.model_path, "test", iteration, cams, gaussians, pipeline, background, kernel_size)
File "extract_mesh_tsdf.py", line 67, in tsdf_fusion
vbg.integrate(frustum_block_coords, o3d_depth, o3d_color, intrinsic,
RuntimeError: [Open3D Error] (void open3d::core::__OPEN3D_CUDA_CHECK(cudaError_t, const char*, int)) /root/Open3D/cpp/open3d/core/CUDAUtils.cpp:310: /root/Open3D/cpp/open3d/core/MemoryManagerCUDA.cpp:25 CUDA runtime error: out of memory
How to solve this problem? Has anyone meet this error?
Hi, we use open3D's GPU version of TSDF fusion and when it is easy to run out of memory when applied to larger scenes. I think you could change the code to use CPU for TSDF fusion. But in general, TSDF results are worse then marching tetrahedral.
Thank you for your reply! Is there a flag in the configuration to switch to CPU for TSDF fusion in the code?
Hi, unfortunately there is no single flag to support cpu TSDF fusion. But you can refer the code here https://github.com/hbb1/2d-gaussian-splatting/blob/main/utils/mesh_utils.py#L156-L180
When I try python extract_mesh_tsdf.py -m exp_TNT/release/Caterpillar --iteration 30000 , I also meet this problem . Then I fix o3d_device = o3d.core.Device('CPU:0'), but it doesn't work. I'm running this code on a high-performance server with A100s and 750G RAM
