MrForExample
CUDA error: no kernel image is available for execution on the device
version of CUDA need match with GPU（hardware）？ CUDA11.7 is match with my GPU. Total VRAM 12288 MB, total RAM 32611 MB pytorch version: 2.3.0+cu121 xformers version: 0.0.26.post1 Set vram state to: NORMAL_VRAM Device: cuda:0 NVIDIA GeForce RTX 3060 : cudaMallocAsync Using xformers cross attention
Comfy3D] [Save_3D_Mesh] Saving model to F:\ComfyUI_3D\ComfyUI\output\Craftsman_Unique3D\Mesh_Gen_0\mesh.obj
100%|████████████████████████████████████████████████████████████████████████████████| 200/200 [00:05<00:00, 34.85it/s]
0%| | 0/100 [00:00<?, ?it/s]
!!! Exception during processing !!! CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ..\c10\cuda\CUDAException.cpp:43 (most recent call first):
00007FFE3E60366200007FFE3E603600 c10.dll!c10::Error::Error [
Traceback (most recent call last):
File "F:\ComfyUI_3D\ComfyUI\execution.py", line 317, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\ComfyUI_3D\ComfyUI\execution.py", line 192, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\ComfyUI_3D\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)
File "F:\ComfyUI_3D\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\ComfyUI_3D\ComfyUI\custom_nodes\ComfyUI-3D-Pack\nodes.py", line 2942, in run_ET_mesh_optimization
vertices, faces = run_mesh_refine(vertices, faces, pil_normal_list, steps=refinement_steps, update_normal_interval=target_update_interval, update_warmup=target_warmup_update_num, )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\ComfyUI_3D\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\Unique3D\mesh_reconstruction\refine.py", line 53, in run_mesh_refine
_, _, target_normal = from_py3d_mesh(multiview_color_projection(py3d_mesh, pils, cameras_list=cameras, weights=[2.0, 0.8, 1.0, 0.8], confidence_threshold=0.1, complete_unseen=False, below_confidence_strategy='original', reweight_with_cosangle='linear'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\ComfyUI_3D\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\Unique3D\scripts\project_mesh.py", line 283, in multiview_color_projection
ret = project_color(meshes, camera, image, eps=eps, resolution=resolution, device=device, use_alpha=use_alpha)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\ComfyUI_3D\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\Unique3D\scripts\project_mesh.py", line 134, in project_color
faces_normals = meshes.faces_normals_packed()[unique_faces]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\ComfyUI_3D\python_embeded\Lib\site-packages\pytorch3d\structures\meshes.py", line 800, in faces_normals_packed
self._compute_face_areas_normals()
File "F:\ComfyUI_3D\python_embeded\Lib\site-packages\pytorch3d\structures\meshes.py", line 868, in _compute_face_areas_normals
face_areas, face_normals = mesh_face_areas_normals(verts_packed, faces_packed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\ComfyUI_3D\python_embeded\Lib\site-packages\torch\autograd\function.py", line 598, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\ComfyUI_3D\python_embeded\Lib\site-packages\pytorch3d\ops\mesh_face_areas_normals.py", line 48, in forward
areas, normals = _C.face_areas_normals_forward(verts, faces)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ..\c10\cuda\CUDAException.cpp:43 (most recent call first):
00007FFE3E60366200007FFE3E603600 c10.dll!c10::Error::Error [
[Comfy3D] ExplicitTarget Mesh Optimization
CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. Exception raised from c10_cuda_check_implementation at ..\c10\cuda\CUDAException.cpp:43 (most recent call first): 00007FF9E1B1366200007FF9E1B13600 c10.dll!c10::Error::Error [
I made it work by uninstalling torch-scatter and reinstalling it from the sources. Not with a downloaded wheel.
pip uninstall torch_scatter pip install torch_scatter
Done in a VS2019 developer prompt with CUDA 12.4 Developer installed.
Hi, right now this project only support CUDA 11.8 and 12.1, because those two are the most widely used versions by researchers and developers
Should be the same with CUDA 12.1, I guess. I had problems building Pytorch3d with CUDA 12.1, I got it to work with CUDA 12.4 so I used that. Everything works for me, except for the 3d Preview.
Same problem with CUDA 12.1. Reinstalling torch_scatter didn't help.
!!! Exception during processing !!! CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\c10\cuda\CUDAException.cpp:43 (most recent call first):
00007FFC82D2462900007FFC82D24580 c10.dll!c10::Error::Error [<unknown file> @ <unknown line number>]
00007FFC82D241DA00007FFC82D24180 c10.dll!c10::detail::torchCheckFail [<unknown file> @ <unknown line number>]
00007FFCEED1046F00007FFCEED10170 c10_cuda.dll!c10::cuda::c10_cuda_check_implementation [<unknown file> @ <unknown line number>]
00007FFC3015FD7100007FFC3010F8D0 _C.cp311-win_amd64.pyd!PyInit__C [<unknown file> @ <unknown line number>]
00007FFC3010BE3700007FFC300F5A90 _C.cp311-win_amd64.pyd!c10::ivalue::Object::operator= [<unknown file> @ <unknown line number>]
00007FFC3011A39E00007FFC3010F8D0 _C.cp311-win_amd64.pyd!PyInit__C [<unknown file> @ <unknown line number>]
00007FFC3011A42400007FFC3010F8D0 _C.cp311-win_amd64.pyd!PyInit__C [<unknown file> @ <unknown line number>]
00007FFC30107FCF00007FFC300F5A90 _C.cp311-win_amd64.pyd!c10::ivalue::Object::operator= [<unknown file> @ <unknown line number>]
00007FFC8EA882BE00007FFC8EA88060 python311.dll!PyObject_MakeTpCall [<unknown file> @ <unknown line number>]
00007FFC8EA8BDDF00007FFC8EA8BB50 python311.dll!PyObject_Vectorcall [<unknown file> @ <unknown line number>]
00007FFC8EA8D42300007FFC8EA8CC20 python311.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFC8EA8AE6400007FFC8EA8ACC0 python311.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFC8EB3507300007FFC8EB35018 python311.dll!PyObject_Call [<unknown file> @ <unknown line number>]
00007FFC5B29134100007FFC5B27DE40 torch_python.dll!THPPointer<_frame>::THPPointer<_frame> [<unknown file> @ <unknown line number>]
00007FFC8EAB75F500007FFC8EAB7540 python311.dll!PyArg_Parse_SizeT [<unknown file> @ <unknown line number>]
00007FFC8EB350C400007FFC8EB35018 python311.dll!PyObject_Call [<unknown file> @ <unknown line number>]
00007FFC8EB34D4000007FFC8EB348EC python311.dll!PyObject_CallObject [<unknown file> @ <unknown line number>]
00007FFC8EA91F7F00007FFC8EA8CC20 python311.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFC8EA8AE6400007FFC8EA8ACC0 python311.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFC8EAD5C2500007FFC8EAD3DC4 python311.dll!PyIter_Send [<unknown file> @ <unknown line number>]
00007FFC8EB34C6A00007FFC8EB348EC python311.dll!PyObject_CallObject [<unknown file> @ <unknown line number>]
00007FFC8EA91F7F00007FFC8EA8CC20 python311.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFC8EA8AE6400007FFC8EA8ACC0 python311.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFC8EB34CF300007FFC8EB348EC python311.dll!PyObject_CallObject [<unknown file> @ <unknown line number>]
00007FFC8EA91F7F00007FFC8EA8CC20 python311.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFC8EA8AE6400007FFC8EA8ACC0 python311.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFC8EAD5C7F00007FFC8EAD3DC4 python311.dll!PyIter_Send [<unknown file> @ <unknown line number>]
00007FFC8EB3507300007FFC8EB35018 python311.dll!PyObject_Call [<unknown file> @ <unknown line number>]
00007FFC8EAF2AA000007FFC8EAF26EC python311.dll!PyInterpreterState_Delete [<unknown file> @ <unknown line number>]
00007FFC8EAF2A0600007FFC8EAF26EC python311.dll!PyInterpreterState_Delete [<unknown file> @ <unknown line number>]
00007FFD3CAD1BB200007FFD3CAD1B20 ucrtbase.dll!configthreadlocale [<unknown file> @ <unknown line number>]
00007FFD3D69737400007FFD3D697360 KERNEL32.DLL!BaseThreadInitThunk [<unknown file> @ <unknown line number>]
00007FFD3F3DCC9100007FFD3F3DCC70 ntdll.dll!RtlUserThreadStart [<unknown file> @ <unknown line number>]
ComfyUI Error Report
Error Details
- Node Type: [Comfy3D] ExplicitTarget Mesh Optimization
- Exception Type: RuntimeError
- Exception Message: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.
Exception raised from c10_cuda_check_implementation at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\c10\cuda\CUDAException.cpp:43 (most recent call first):
00007FFC82D2462900007FFC82D24580 c10.dll!c10::Error::Error [
Stack Trace
File "h:\ComfyUI\ComfyUI\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\ComfyUI\ComfyUI\execution.py", line 198, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\ComfyUI\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)
File "h:\ComfyUI\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-3D-Pack\nodes.py", line 2955, in run_ET_mesh_optimization
vertices, faces = run_mesh_refine(vertices, faces, pil_normal_list, steps=refinement_steps, update_normal_interval=target_update_interval, update_warmup=target_warmup_update_num, )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\Unique3D\mesh_reconstruction\refine.py", line 53, in run_mesh_refine
_, _, target_normal = from_py3d_mesh(multiview_color_projection(py3d_mesh, pils, cameras_list=cameras, weights=[2.0, 0.8, 1.0, 0.8], confidence_threshold=0.1, complete_unseen=False, below_confidence_strategy='original', reweight_with_cosangle='linear'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\Unique3D\scripts\project_mesh.py", line 282, in multiview_color_projection
ret = project_color(meshes, camera, image, eps=eps, resolution=resolution, device=device, use_alpha=use_alpha)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\Unique3D\scripts\project_mesh.py", line 135, in project_color
faces_normals = meshes.faces_normals_packed()[unique_faces]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\ComfyUI\python_embeded\Lib\site-packages\pytorch3d\structures\meshes.py", line 800, in faces_normals_packed
self._compute_face_areas_normals()
File "h:\ComfyUI\python_embeded\Lib\site-packages\pytorch3d\structures\meshes.py", line 868, in _compute_face_areas_normals
face_areas, face_normals = mesh_face_areas_normals(verts_packed, faces_packed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\ComfyUI\python_embeded\Lib\site-packages\torch\autograd\function.py", line 574, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\ComfyUI\python_embeded\Lib\site-packages\pytorch3d\ops\mesh_face_areas_normals.py", line 48, in forward
areas, normals = _C.face_areas_normals_forward(verts, faces)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
System Information
- ComfyUI Version: v0.2.2-72-g3a0eeee
- Arguments: ComfyUI\main.py --windows-standalone-build
- OS: nt
- Python Version: 3.11.9 (tags/v3.11.9:de54cf5, Apr 2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
- Embedded Python: true
- PyTorch Version: 2.4.0+cu121
Devices
- Name: cuda:0 NVIDIA GeForce RTX 2080 Ti : cudaMallocAsync
- Type: cuda
- VRAM Total: 11810701312
- VRAM Free: 4405207018
- Torch VRAM Total: 11341398016
- Torch VRAM Free: 4405207018
Logs
2024-09-23 16:55:08,505 - root - INFO - Total VRAM 11264 MB, total RAM 130939 MB
2024-09-23 16:55:08,505 - root - INFO - pytorch version: 2.4.0+cu121
2024-09-23 16:55:10,354 - root - INFO - xformers version: 0.0.27.post2
2024-09-23 16:55:10,365 - root - INFO - Set vram state to: NORMAL_VRAM
2024-09-23 16:55:10,366 - root - INFO - Device: cuda:0 NVIDIA GeForce RTX 2080 Ti : cudaMallocAsync
2024-09-23 16:55:10,614 - root - INFO - Using xformers cross attention
2024-09-23 16:55:12,486 - root - INFO - [Prompt Server] web root: h:\ComfyUI\ComfyUI\web
2024-09-23 16:55:15,656 - py.warnings - WARNING - h:\ComfyUI\python_embeded\Lib\site-packages\kiui\nn\__init__.py:31: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
2024-09-23 16:55:15,656 - py.warnings - WARNING - h:\ComfyUI\python_embeded\Lib\site-packages\kiui\nn\__init__.py:37: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
@torch.cuda.amp.custom_bwd
2024-09-23 16:55:15,721 - py.warnings - WARNING - H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\TriplaneGaussian\utils\ops.py:38: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
@custom_fwd(cast_inputs=torch.float32)
2024-09-23 16:55:15,721 - py.warnings - WARNING - H:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-3D-Pack\Gen_3D_Modules\TriplaneGaussian\utils\ops.py:44: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
@custom_bwd
Attached Workflow
Please make sure that workflow does not contain any sensitive information such as API keys or passwords.
Default Unique3D_All_Stages.json
Update: Installing manually proper pytorch3d wheel from https://miropsota.github.io/torch_packages_builder/pytorch3d/ (for my case it's a pytorch3d-0.7.8+pt2.4.0cu121-cp311-cp311-win_amd64.whl) and all works
Update: Installing manually proper pytorch3d wheel from https://miropsota.github.io/torch_packages_builder/pytorch3d/ (for my case it's a pytorch3d-0.7.8+pt2.4.0cu121-cp311-cp311-win_amd64.whl) and all works
thank you! you solve my problem
This was a pain. I'm still not 100% how much of what I did fixed it, I built torch_scatter from source, I built nvdiffrast from source, I built pytorch3d from source, I then went into the original Hunyuan 3D-2 repository and built the custom_rasterizer from source (since that 3D generator was originally the one I wanted to use). At the very end of it, I am pretty sure that it came down to was that the wheels I was using were not the built against the same GPU architecture as my own gpu, I was using an A6000 for this so that may be why I had so many problems because I bet most people building this are using like 4000 or 3000 series or something. But I set the TORCH_CUDA_ARCH_LIST to 8.6 (because that is what my gpu architecture is, you'd probably have to look up yours if you have the same issue) and made sure it built against that and this seemed to solve the issue...
although my 3d previews are still not working, I have to use the 3D preview beta node that comes with comfyui now in order to see anything... which unfortunately means I have to actually save the model in order to view it.
