RuntimeError: r.nvmlDeviceGetNvLinkRemoteDeviceType_ INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":27, please report a bug to PyTorch. Can't find nvmlDeviceGetNvLinkRemoteDeviceType: /lib64/libnvidia-ml.so.1: undefined symbol: nvmlDeviceGetNvLinkRemoteDeviceType

[HunYuanfeng]

trafficstars

When I want to use a for loop like this to run multiple results:

pipe = CogVideoXPipeline.from_pretrained(...)
for prompt in prompts:
    pipe.enable_sequential_cpu_offload()
    prompt_embeds, _ = pipe.encode_prompt(prompt=prompt,...)
    video = pipe(...).frames[0]
    export_to_video(...)

it gives an error like below:

Traceback (most recent call last):
  File "/group/40032/ych/projects/my_cogvideox/CogVideo/inference/my_cli_demo.py", line 162, in <module>
    generate_multi_videos(
  File "/group/40032/ych/projects/my_cogvideox/CogVideo/inference/my_cli_demo.py", line 48, in generate_multi_videos
    prompt_embeds, _ = pipe.encode_prompt(
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/diffusers/pipelines/cogvideo/pipeline_cogvideox.py", line 279, in encode_prompt
    prompt_embeds = self._get_t5_prompt_embeds(
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/diffusers/pipelines/cogvideo/pipeline_cogvideox.py", line 222, in _get_t5_prompt_embeds
    prompt_embeds = self.text_encoder(text_input_ids.to(device))[0]
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/accelerate/hooks.py", line 169, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1971, in forward
    encoder_outputs = self.encoder(
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1106, in forward
    layer_outputs = layer_module(
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 746, in forward
    hidden_states = self.layer[-1](hidden_states)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 335, in forward
    forwarded_states = self.DenseReluDense(forwarded_states)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 318, in forward
    hidden_states = self.wo(hidden_states)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/accelerate/hooks.py", line 164, in new_forward
    args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/accelerate/hooks.py", line 354, in pre_forward
    set_module_tensor_to_device(
  File "/data/miniconda3/envs/CogVideo/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 416, in set_module_tensor_to_device
    new_value = value.to(device)
RuntimeError: r.nvmlDeviceGetNvLinkRemoteDeviceType_ INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":27, please report a bug to PyTorch. Can't find nvmlDeviceGetNvLinkRemoteDeviceType: /lib64/libnvidia-ml.so.1: undefined symbol: nvmlDeviceGetNvLinkRemoteDeviceType

How could I solve this problem?

[zRzRzRzRzRzRzR]

这个是依赖的问题吧

[Tegmier]

有什么进展吗？

[Meta-YZ]

我也遇到这个问题了，请问有解决方案吗