RuntimeError: GET was unable to find an engine to execute this computation

[quarterturn]

trafficstars

System Info / 系統信息

declare -x CONDA_DEFAULT_ENV="cog" declare -x CONDA_EXE="/opt/conda/bin/conda" declare -x CONDA_PREFIX="/home/alex/.conda/envs/cog" declare -x CONDA_PREFIX_1="/opt/conda" declare -x CONDA_PROMPT_MODIFIER="(cog) " declare -x CONDA_PYTHON_EXE="/opt/conda/bin/python" declare -x CONDA_SHLVL="2" declare -x CUDA_HOME="/usr/local/cuda-12.5/"

Python 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import torch print(torch.version.cuda) 12.4 exit()

Python 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import torch available_gpus = [torch.cuda.device(i) for i in range(torch.cuda.device_count())] available_gpus [<torch.cuda.device object at 0x7f4185909850>, <torch.cuda.device object at 0x7f42c7de8f90>, <torch.cuda.device object at 0x7f42c7de9250>, <torch.cuda.device object at 0x7f42c7de9290>, <torch.cuda.device object at 0x7f42c7de9310>]

nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Thu_Jun__6_02:18:23_PDT_2024 Cuda compilation tools, release 12.5, V12.5.82 Build cuda_12.5.r12.5/compiler.34385749_0

nvidia-smi Fri Aug 30 13:36:58 2024
+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 555.42.06 Driver Version: 555.42.06 CUDA Version: 12.5 |

Information / 问题信息

• [X] The official example scripts / 官方的示例脚本
• [ ] My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

python3 cli_demo.py --model_path CogVideoX-5b/ --prompt "A garden comes to life as a kaleidoscope of butterflies flutters amidst the blossoms, their delicate wings casting shadows on the petals below. In the background, a grand fountain cascades water with a gentle splendor, its rhythmic sound providing a soothing backdrop. Beneath the cool shade of a mature tree, a solitary wooden chair invites solitude and reflection, its smooth surface worn by the touch of countless visitors seeking a moment of tranquility in nature's embrace."

Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.19s/it] Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████| 5/5 [00:08<00:00, 1.79s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [09:04<00:00, 10.90s/it] Traceback (most recent call last): File "/home/quarterturn/CogVideo/inference/cli_demo.py", line 106, in generate_video( File "/home/quarterturn/CogVideo/inference/cli_demo.py", line 66, in generate_video video = pipe( ^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/pipelines/cogvideo/pipeline_cogvideox.py", line 735, in call video = self.decode_latents(latents) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/pipelines/cogvideo/pipeline_cogvideox.py", line 360, in decode_latents frames = self.vae.decode(latents).sample ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper return method(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 1153, in decode decoded = self._decode(z).sample ^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 1123, in _decode z_intermediate = self.decoder(z_intermediate) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/accelerate/hooks.py", line 169, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 851, in forward hidden_states = self.conv_in(sample) ^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/accelerate/hooks.py", line 169, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 144, in forward output = self.conv(inputs) ^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/accelerate/hooks.py", line 169, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 64, in forward return super().forward(input) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 608, in forward return self._conv_forward(input, self.weight, self.bias) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/quarterturn/.conda/envs/cog/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 603, in _conv_forward return F.conv3d( ^^^^^^^^^ RuntimeError: GET was unable to find an engine to execute this computation

Expected behavior / 期待表现

Should produce video.

[zRzRzRzRzRzRzR]

are cuda_12.5 compile with torch? abd what is your GPU