MrForExample
Out of Memory on a 24G GPU Card from InstantMesh
Hello, I'm getting memory allocation error with every workflow that involves this instant-mesh reconstruction node. I'm running on a machine with 24G gpu memory. Should this happen? How could I fix it?
Traceback (most recent call last):
File "/home/zling/Documents/projects/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/home/zling/Documents/projects/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/home/zling/Documents/projects/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/home/zling/anaconda3/envs/comfyui/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/nodes.py", line 2272, in run_LRM
mesh_out = lrm_model.extract_mesh(
File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/instant_mesh/models/lrm_mesh.py", line 357, in extract_mesh
mesh_v, mesh_f, sdf, deformation, v_deformed, sdf_reg_loss = self.get_geometry_prediction(planes)
File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/instant_mesh/models/lrm_mesh.py", line 165, in get_geometry_prediction
sdf, deformation, sdf_reg_loss, weight = self.get_sdf_deformation_prediction(planes)
File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/instant_mesh/models/lrm_mesh.py", line 110, in get_sdf_deformation_prediction
sdf, deformation, weight = torch.utils.checkpoint.checkpoint(
File "/home/zling/anaconda3/envs/comfyui/lib/python3.8/site-packages/torch/_compile.py", line 24, in inner
return torch._dynamo.disable(fn, recursive)(*args, **kwargs)
File "/home/zling/anaconda3/envs/comfyui/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn
return fn(*args, **kwargs)
File "/home/zling/anaconda3/envs/comfyui/lib/python3.8/site-packages/torch/_dynamo/external_utils.py", line 36, in inner
return fn(*args, **kwargs)
File "/home/zling/anaconda3/envs/comfyui/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 494, in checkpoint
ret = function(*args, **kwargs)
File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/instant_mesh/models/renderer/synthesizer_mesh.py", line 132, in get_geometry_prediction
sdf, deformation, weight = self.decoder.get_geometry_prediction(sampled_features, flexicubes_indices)
File "/home/zling/Documents/projects/ComfyUI/custom_nodes/ComfyUI-3D-Pack/instant_mesh/models/renderer/synthesizer_mesh.py", line 76, in get_geometry_prediction
grid_features = torch.index_select(input=sampled_features, index=flexicubes_indices.reshape(-1), dim=1)
torch.cuda.OutOfMemoryError: Allocation on device
Hi all,
I'm getting the same error.
I noticed that if I try running the pipeline again immediately afterwards it completes without any issues, however. Is this the case on your end as well @SergioKingOne and @iszihan ?
Also, if you scroll to the beginning of the ComyUI console do you see a message that says: "UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']."
I noticed this message appeared immediately after installing the ComfyUI 3D Pack. I'm wondering if setting this value correctly will resolve the issue, but I don't know what to set it to.
Hi @tcarneyjohnson. That env variable is for setting the available compute capability architectures you can run. That should not be a problem.
A workaround for me was to directly run instant mesh without ComfyUI (i.e. download instant mesh repo and run it with a command).
Another possible solution is to unload the MVs model once its done its job but I could not achieve it.
This also happens to me on the "Triplane_Gaussian_Transformers_to_3DGS_and_3DMesh.json" workflow.
I'm running a 3080 series 24gb vram which is minimum listed req for gaussian to marching cubes. Everything else I use works but the 3d marching cubes.
It crashes on this node:
I get the same error:
File "D:\Code\ComfyUI_windows_portable\python_embeded\Lib\site-packages\diff_gaussian_rasterization\__init__.py", line 92, in forward
num_rendered, color, depth, alpha, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Allocation on device
I get the same warning:
"UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']."
Fixed this by reinstalling diff-gaussian-rasterization
pip uninstall diff_gaussian_rasterization
pip install "git+https://github.com/ashawkey/diff-gaussian-rasterization.git"
Fixed this by reinstalling diff-gaussian-rasterization
pip uninstall diff_gaussian_rasterizationpip install "git+https://github.com/ashawkey/diff-gaussian-rasterization.git"
using the python.exe embedded or ?? Inside the custom nodes folder? Worked for me thanks.
This is also occurring for me on any workflow that uses the InstantMesh node. Here's the callstack from running the Era3D_to_InstantMesh workflow:
[Comfy3D] [Load_InstantMesh_Reconstruction_Model] loaded model ckpt from D:\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Checkpoints\InstantMesh\instant_mesh_large.ckpt
!!! Exception during processing !!! Allocation on device
Traceback (most recent call last):
File "D:\ComfyUI\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\execution.py", line 198, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)
File "D:\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\JeffSipko\miniconda3\envs\py311\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\custom_nodes\ComfyUI-3D-Pack\nodes.py", line 2496, in run_LRM
mesh_out = lrm_model.extract_mesh(
^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\InstantMesh\models\lrm_mesh.py", line 357, in extract_mesh
mesh_v, mesh_f, sdf, deformation, v_deformed, sdf_reg_loss = self.get_geometry_prediction(planes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\InstantMesh\models\lrm_mesh.py", line 165, in get_geometry_prediction
sdf, deformation, sdf_reg_loss, weight = self.get_sdf_deformation_prediction(planes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\InstantMesh\models\lrm_mesh.py", line 110, in get_sdf_deformation_prediction
sdf, deformation, weight = torch.utils.checkpoint.checkpoint(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\JeffSipko\miniconda3\envs\py311\Lib\site-packages\torch\_compile.py", line 31, in inner
return disable_fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\JeffSipko\miniconda3\envs\py311\Lib\site-packages\torch\_dynamo\eval_frame.py", line 600, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\JeffSipko\miniconda3\envs\py311\Lib\site-packages\torch\utils\checkpoint.py", line 488, in checkpoint
ret = function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\InstantMesh\models\renderer\synthesizer_mesh.py", line 132, in get_geometry_prediction
sdf, deformation, weight = self.decoder.get_geometry_prediction(sampled_features, flexicubes_indices)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\InstantMesh\models\renderer\synthesizer_mesh.py", line 76, in get_geometry_prediction
grid_features = torch.index_select(input=sampled_features, index=flexicubes_indices.reshape(-1), dim=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: Allocation on device
My configuration from Comfy startup:
## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2024-11-25 13:14:39.768854
** Platform: Windows
** Python version: 3.11.9 | packaged by Anaconda, Inc. | (main, Apr 19 2024, 16:40:41) [MSC v.1916 64 bit (AMD64)]
** Python executable: C:\Users\sipkode\miniconda3\envs\py311\python.exe
** ComfyUI Path: D:\ComfyUI
** Log path: D:\ComfyUI\comfyui.log
Prestartup times for custom nodes:
1.1 seconds: D:\ComfyUI\custom_nodes\ComfyUI-Manager
Total VRAM 24564 MB, total RAM 65254 MB
pytorch version: 2.4.0
xformers version: 0.0.27.post2
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
Using xformers cross attention
Re-installing diff_gaussian_rasterization as suggested above did not work...
edit: Same result in LOW_VRAM mode...
Fixed this by reinstalling diff-gaussian-rasterization
pip uninstall diff_gaussian_rasterizationpip install "git+https://github.com/ashawkey/diff-gaussian-rasterization.git"
Could you help please, I don't understand where exactly to enter these commands. Wherever I tried it says that it is not installed (diff-gaussian-rasterization)
Fixed this by reinstalling diff-gaussian-rasterization
pip uninstall diff_gaussian_rasterizationpip install "git+https://github.com/ashawkey/diff-gaussian-rasterization.git"Could you help please, I don't understand where exactly to enter these commands. Wherever I tried it says that it is not installed (diff-gaussian-rasterization)
@TRGLN if you are using ComfyUI portable version with Python embedded you have to do the following:
- Open a PowerShell window: click on Window's Start button > type
powershell> click on "Windows PowerShell" - Navigate to your ComfyUI folder by typing the command:
cd D:\update_with_your_path\ComfyUI\python_embeded - Execute the commands
.\python.exe -s -m pip uninstall diff_gaussian_rasterization.\python.exe -s -m pip install "git+https://github.com/ashawkey/diff-gaussian-rasterization.git"
When i get to an InstantMesh Reconstruction Moodel I also getting this error Running on an nvidea a5000 24gb card
I have tried the reinstall of diff-gaussian-rasterization but that didnt help here
Fetching 8 files: 100%|██████████████████████████████████████████████████████████████████| 8/8 [00:38<00:00, 4.79s/it]
Loading pipeline components...: 22%|███████████▌ | 2/9 [00:00<00:00, 13.81it/s]The config attributes {'decay': 0.9999, 'inv_gamma': 1.0, 'min_decay': 0.0, 'optimization_step': 40000, 'power': 0.6666666666666666, 'update_after_step': 0, 'use_ema_warmup': False} were passed to UNetMV2DConditionModel, but are not expected and will be ignored. Please verify your config.json configuration file.
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 9/9 [00:00<00:00, 22.59it/s]
Expected types for feature_extractor: (<class 'transformers.models.clip.feature_extraction_clip.CLIPFeatureExtractor'>,), got <class 'transformers.models.clip.image_processing_clip.CLIPImageProcessor'>.
Expected types for unet: (<class 'diffusers.models.unets.unet_2d_condition.UNet2DConditionModel'>,), got <class 'Era3D.mvdiffusion.models.unet_mv2d_condition.UNetMV2DConditionModel'>.
FETCH ComfyRegistry Data [DONE]
[ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json [DONE]
[ComfyUI-Manager] All startup tasks have been completed.
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:21<00:00, 1.84it/s]
[Comfy3D] [Load_InstantMesh_Reconstruction_Model] loaded model ckpt from C:\Users\va-ai.render\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Checkpoints\InstantMesh\instant_mesh_large.ckpt
!!! Exception during processing !!! Allocation on device
Traceback (most recent call last):
File "C:\Users\va-ai.render\ComfyUI\execution.py", line 361, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\ComfyUI\execution.py", line 236, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\ComfyUI\execution.py", line 208, in _map_node_over_list
process_inputs(input_dict, i)
File "C:\Users\va-ai.render\ComfyUI\execution.py", line 197, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\ComfyUI\custom_nodes\ComfyUI-3D-Pack\nodes.py", line 2539, in run_LRM
mesh_out = lrm_model.extract_mesh(
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\InstantMesh\models\lrm_mesh.py", line 357, in extract_mesh
mesh_v, mesh_f, sdf, deformation, v_deformed, sdf_reg_loss = self.get_geometry_prediction(planes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\InstantMesh\models\lrm_mesh.py", line 165, in get_geometry_prediction
sdf, deformation, sdf_reg_loss, weight = self.get_sdf_deformation_prediction(planes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\InstantMesh\models\lrm_mesh.py", line 110, in get_sdf_deformation_prediction
sdf, deformation, weight = torch.utils.checkpoint.checkpoint(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\_compile.py", line 51, in inner
return disable_fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\_dynamo\eval_frame.py", line 838, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\utils\checkpoint.py", line 495, in checkpoint
ret = function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\InstantMesh\models\renderer\synthesizer_mesh.py", line 132, in get_geometry_prediction
sdf, deformation, weight = self.decoder.get_geometry_prediction(sampled_features, flexicubes_indices)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\va-ai.render\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\InstantMesh\models\renderer\synthesizer_mesh.py", line 76, in get_geometry_prediction
grid_features = torch.index_select(input=sampled_features, index=flexicubes_indices.reshape(-1), dim=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: Allocation on device
