Cuda Out of Memory Error/ Support for Multi GPU's

[srikar242]

Hello @JustinPack @yxgeee @bluestyle97 @eltociear I am getting cuda out of memory error when trying to run this model on my device using command line. I have 2GPU's each of 16GB. Issue here is that it is using only 1 GPU and its not accessing the second one. How can I make the pipeline use multiple GPU's? I ran run.py script using command line without rmb. Still I am getting the issue. Command used: python run.py configs/instant-mesh-large.yaml examples/hatsune_miku.png --save_video

Error: Traceback (most recent call last): File "/home/jupyter/InstantMesh/run.py", line 221, in <module> mesh_out = model.extract_mesh( File "/home/jupyter/InstantMesh/src/models/lrm_mesh.py", line 357, in extract_mesh mesh_v, mesh_f, sdf, deformation, v_deformed, sdf_reg_loss = self.get_geometry_prediction(planes) File "/home/jupyter/InstantMesh/src/models/lrm_mesh.py", line 165, in get_geometry_prediction sdf, deformation, sdf_reg_loss, weight = self.get_sdf_deformation_prediction(planes) File "/home/jupyter/InstantMesh/src/models/lrm_mesh.py", line 110, in get_sdf_deformation_prediction sdf, deformation, weight = torch.utils.checkpoint.checkpoint( File "/opt/conda/envs/instantmesh/lib/python3.10/site-packages/torch/_compile.py", line 24, in inner return torch._dynamo.disable(fn, recursive)(*args, **kwargs) File "/opt/conda/envs/instantmesh/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/instantmesh/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner return fn(*args, **kwargs) File "/opt/conda/envs/instantmesh/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 458, in checkpoint ret = function(*args, **kwargs) File "/home/jupyter/InstantMesh/src/models/renderer/synthesizer_mesh.py", line 132, in get_geometry_prediction sdf, deformation, weight = self.decoder.get_geometry_prediction(sampled_features, flexicubes_indices) File "/home/jupyter/InstantMesh/src/models/renderer/synthesizer_mesh.py", line 76, in get_geometry_prediction grid_features = torch.index_select(input=sampled_features, index=flexicubes_indices.reshape(-1), dim=1) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 15.00 GiB. GPU 0 has a total capacty of 14.58 GiB of which 10.58 GiB is free. Including non-PyTorch memory, this process has 3.99 GiB memory in use. Of the allocated memory 3.63 GiB is allocated by PyTorch, and 226.96 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[nattybones]

Check out my fork: https://github.com/nattybones/InstantMesh2gpu

I have InstantMesh running on two 24GB GPus by splitting the functions. Claude3 wrote the script.

[TzyTman]

Check out my fork: https://github.com/nattybones/InstantMesh2gpu

I have InstantMesh running on two 24GB GPus by splitting the functions. Claude3 wrote the script. Use your project to run up and report errors!

[bluestyle97]

We have add two-gpu support. Please checkout the newest code!